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Abstract. Macdonald superpolynomials provide a remarkably rich generalization of the usual Macdonald polynomials. 
The starting point of this work is the observation of a previously unnoticed stability property of the Macdonald superpoly- 
nomials when the fermionic sector m is sufficiently large; their decomposition in the monomial basis is then independent 
of m. These stable superpolynomials are readily mapped into bisymmetric polynomials, an operation that spoils the ring 
structure but drastically simplifies the associated vector space. Our first main result is a factorization of the (stable) 
bisymmetric Macdonald polynomials, called double Macdonald polynomials and indexed by pairs of partitions, into a 
product of Macdonald polynomials (albeit subject to non-trivial plethystic transformations). As an off-shoot, we note 
that, after multiplication by a t-Vandermonde determinant, this provides explicit formulas for a huge class of Macdonald 
■ polynomials with prescribed symmetry. The double (plethystically-)deformed Macdonald polynomials are then shown 

to have a positive expansion in the corresponding bisymmetric Schur basis (the expansion coefficients being generalized 
Kostka's). Two points of contact with the hyperoctahedral group are then uncovered. First, the q = t = 1 specialization 
of the Kostka coefficients corresponds to the dimensions of the irreducible representations of the hyperoctahedral group. 
Second, the action of a generalized Nabla operator on the bisymmetric Schur polynomial is found to be Schur positive, 
with a possible interpretation as that of the Frobenius series of a certain bigraded module of dimension (2n -|- 1)", 
a formula characteristic of the Coxeter group of type Bn- Unfortunately, there seems to be no relation between the 
double Macdonalds and a bigraded (Garsia-Haiman-like) submodule thereof. On the other hand, the aforementioned 
factorization of the double Macdonald polynomials leads immediately to the generalization of basically every elementary 
properties of the Macdonald polynomials to the double case (norm, kernel, duality, evaluation, etc). When lifted back to 
superspace, these results provide proofs of various previously formulated conjectures in the stable regime. 



1. Introduction 

1.1. From superpolynomials to bisymmetric polynomials. The Macdonald polynomials in superspace, which 
have been recently introduced in [6, 7 , provide a conibinatorially-rich generalization of the Macdonald polynomials. 
We show in this article that when the fermionic sector is large enough, the Macdonald polynomials in superspace 
embody a natural form of a bisymmetric extension of the Macdonald polynomials whose corresponding Kostka and 
Nabla combinatorics is that of the hyperoctahedral group Bn (whereas it is that of the symmetric group Sn in the 
usual case). This novel version of the Macdonald polynomials will be referred to, for short, as the double Macdonald 
polynomials Q 

Let us put these statements in context by first recalling the definition of a superpolynomial or equivalently said, 
a polynomial in superspace. In addition to be a polynomial in the usual indeterniinates xi, ■ ■ ■ ,xn over Q(q,t), it 
is a function of N anticommuting (or fermionic) variables 6*1, •• • ,dN, with 6f — 0. Such polynomials are said to be 
symmetric if they are invariant with respect to the simultaneous interchange of {xi,9i) 4-> (xj,9j). Due to the latter 
property, if a superpolynomial is symmetric, it is always a sum of monomials Oi-^ ■ ■ ■ Oi^ multiplied by a polynomial 
antisymmetric in the variables Xi^ , • • • , Xi^ and symmetric in the remaining ones. The superpolynomials we consider 
are always bi-homogeneous in the variables x and 9, and their fermionic degree (the degree in 9) will be denoted m. 
Finally, a symmetric superpolynomial is indexed by a superpartition A: a pair of partitions (A"; A'') such that A" 
has m distinct parts (the m-th one being allowed to be 0). The partitions A" (resp. A^) captures the degree of the 
antisymmetric (resp. symmetric) polynomial associated to each term 9i-^ ■ ■ ■ 9i^^ of the superpolynomial. 



^The standard notation for the hyperoctahedral group being B„, it should however be kept in mind that these are not the Macdonald 
polynomials defined on the root lattice of type B |20l 1211 . These are also different from the Macdonald polynomials with hyperoctahedral 
symmetries constructed in ||11| . 
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To make these comments more concrete, consider the generahzation of the monomial and power-sum basis to super- 
space [6]: 

creSN 

(where the prime indicates a sum over distinct terms) and 

N N 

PA = PAi ■ ■ ■ PA„PA„+i ■ ■ ■ PAjv , where = ^ Oix'l and = ^ a;^ (1.2) 

i=l i=l 

(with fc > and r > 1). If = 2 and = 3 we have for instance: 

"i(3,l;l) = 9i92{xlx2 - xlxi)x3 + 6'i6'3 (xj^s - xlxi)x2 + 6*2613 (2:3X3 - xlx2)xi (1.3) 

and 

P(3,i;i) = (^12^1 + ^22:2 + OsxDiOiXi + 62X2 + 03X3){xi +X2+ X3) 

= 9i02{x\x2 - xlxi){xi +X2 + x:i) + Oid^{x\x:i - a;3a;i)(a;i +X2 + x^i) -f 6'26'3(x3a;3 - xlx2){xi + X2 + x^). 

(1.4) 

To any superpolynomial, we can associate a bisymmetric polynomial. For a superpolynomial of fermionic degree 
m, this is done by (1)- extracting the coefficient of 0i • • • 9m of the superpolynomial, and (2)-dividing the result by 
the Vandermonde determinant in the variables xi, • • • , Xm- By construction, the resulting polynomial is symmetric in 
both the variables x = (xi, • • • , Xm) and y — {x^+i^ ■ • • , xiq). For the examples presented above, this procedure yields 
{x\x2 + xiX2)x^ and {x\x2 + xix'^){xi -I- 2:2 -f 2:3) respectively. 

When, as was just described, we pass to bisymmetric polynomials, the natural ring structure of the space of super- 
polynomials is lost (the ring structure of the space of bisymmetric polynomials is not the right one, especially when 
considering connections with supersymmetry [SJITU]). However, in this article, we are only interested in the underlying 
vector space. As such, it will prove more convenient to work with bisymmetric polynomials. We will then have at our 
disposal the very powerful (and much better known) language of symmetric function theory. 

A bisymmetric polynomial is naturally indexed by a pair of partitions extracted from the superpartition. To be more 
precise, the correspondence between a superpartition of fermionic degree m and a pair of partitions A, ^ is 

A = (A"^; A") o (A, ^) = (A"^ - (5™, A'*), (1.5) 

where J"* := (m — 1, ... ,0) stands for the staircase partition. Observe that from (jl.Sp . it is immediate that ^(A) < m. 
With this correspondence, the monomial and power-sums symmetric superpolynomials are then associated to the 
following bisymmetric polynomials respectively (see Appendix |X] for a detailed derivation of this correspondence) 

TTiA i — > mx^f^{x,y) -.^ sx{x)mf_,{y), (1.6) 

and 

PA < — > px^f,{x,y) := sx{x)pf,{x,y) , (1.7) 
where s\, m\, and p\ are respectively the Schur, monomial and power-sum symmetric functions. In the case of rn\^^, 
the functions in the product depend upon complementary sets of variables, while pA,/i does not {p^{x,y) is the usual 
power-sum symmetric functions in the union of the variables x and y). In the two cases, it is still noteworthy that the 
corresponding bisymmetric polynomials have such a simple factorization (we stress that m\ and pA are both functions 
of the variables xi, - ■ ■ ,xn,Oi, - ■ ■ , On)- 

The bisymmetric polynomials associated to the Macdonald superpolynomials PA(a;i, • • • , xn, di, - ■ ■ , 9m] q,t) |6l [7] 
will simply be denoted Pa(x, y; g, t) For reasons to become clear shortly, it is more convenient at this point to keep 
using superpartitions as indices even for the bisymmetric version of the Macdonald superpolynomials (A captures for 
instance the information on the fermionic degree m which corresponds to the maximal length A can have). We now 



The meaning of Pa will be clear from the context and actually, in the body of the text, it will always refer to the bisymmetric version. 
The two forms are distinguished by their explicit variable-dependence: either {x,8) or {x,y). 
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translate in the language of bisymmetric functions Theorem 1 of ^ which establishes the existence of the Macdonald 
superpolynomials. It relies on the dominance ordering on superpartitions, which is defined as 

A>n iff A* >n* and A® > n® (1.8) 

where the order on partitions is the dominance ordering, and where 

A* = (A + 5"') U M and A® = (A + 5"'+'^) U fi . (1.9) 

Theorem 1. Given a superpartition A 4-> (A,/i) and N ~~ m > |A| + \fi\, there exists a unique bisymmetric polynomial 
P\ = Pa^x, y; q, t), with x = (xi, . . . , Xm) and y = {xm+i, • ■ • , xn), such that: 

1) Pa ~ "nix + lower terms, 

(1.10) 

2) ((PA,Po))g,t = if A^n 

The dominance ordering on pairs of partitions is such that (A,/i) > {i-u,ri) iff the corresponding superpartitions A and 
o {uj, f]) are such that A>Q.. The scalar product is defined on the power-sums (|1.7|) as0 

((PA,M, Pu:,n))q,t = Sx^6fj,jjq^^^ z,,{q,t) , (1.11) 

where Zfi{q,t) is given in (|2.5p . 

We stress that the monomial expansion of Pa is independent oi N — m (granted that — m is large enough) and 
thus N — m can be considered infinite. Unexpectedly, a similar independence upon m holds. Let us denote by n the 
total degree of the pair (A, /i), that is, 

n=\X\ + \^^\. (1.12) 

In the previous theorem, the restriction £(X) < m in the correspondence (jl.Sp is lifted when m > n. What is remarkable, 
and totally unexpected from the supersymmetric point of view, is that even though the ordering (|1.8|) appears to be 
highly dependant of m, the monomial expansion (|1.10|) of the bisymmetric polynomial Pa does not depend on m 
whenever m > n. This will be referred to as the stable sector of the bisymmetric Macdonald polynomials. This 
phenomenon is described schematically in Figure [OI Let us illustrate the stability property of Pa, where A -H> (0, (2)), 
by displaying the monomial decompositions when n = 2 for four different values of m: 

(l-i)(l + (j) (l-t) 

P{m = ™0,(2) + — Y^t — + ^w.w 

, (l-t)(l + gt) 

^(l,0;2) - "^0,(2) + l_qt2 "^0,(1,1) 

(1 - <) (1 + qt) 
^(2,i.0;2) - m^,{2} + i_qt2 "^0,(1,1) 

(1 - <) (1 + qt) , . 

-P(3,2,l:0;2) = "^0,(2) H l^T^ "^0,(1,1) (1-13) 

The first case, for which m = 1 < 2 — n, does not belong to the stable sector. In the other three cases, one recovers 
three identical expressions (even though the corresponding monomials depend on different sets of variables). 

In the stable sector, it will thus be more natural to index the bisymmetric polynomial Pa '■— P\,fj. by the pair of 
partitions (A, /i), even more so that in this sector the ordering (jl.Sp can be replaced by the following dominance ordering 
on pairs of partitions (cf. Proposition [32l in Appendix IB)) : for (A,/x) and (aj,??) both of total degree n, 

(A,^)>(w,ry) iff Ai H h A^ > wi H V uji and |A|+/iiH h > |w| + ryi H h?7j, Vi,j (1.14) 

where it is understood that A^ = if i > ^(A) (and similarly for /i,a; and r/). 

Because they are labelled by two partitions and, as we will see shortly, they are naturally viewed as a function of 
two set of (commuting) variables, the bisymmetric Macdonald polynomials in the stable sector will be called double 
Macdonald polynomials. Therefore, in the stable sector Theorem [T] becomes: 



■^In the expression of the scalar product, we have dropped a factor l/g)™''" ^'/■^ which does not affect the orthogonality. 
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Figure 1 . To each dot corresponds a family of superpolynomials indexed by superpartitions of total 
degree |A°| + |A^| = n + m{m— l)/2 and whose component A° has exactly m parts. The sector m>n 
(indicated in gray) is the domain in which the bisymnietric polynomials are stable. In other words, for 
a fixed value of m, all the points below and on the diagonal represent equivalent families, for which 
convenient representatives are the points on the diagonal (larger dots). In the complementary region, 
the superspace formalism (via the dominance ordering for superpartitions) is mandatory. 

n 

7 

6 
5 
4 
3 
2 
1 


01234567 rn 




Theorem 2. Let (A,/i) be of total degree n. Then the double Macdonald polynomials P\,f^(x,y; q,t) , where x — 
{xi, . . . , Xm) and y — {xm+i, ■ ■ ■ , xn) (with ni, N — m > n) are the unique bisymmetric polynomials such that 

P\,,,(x,y;q,t) ^ mx^f,{x,y) + ^ cx.t,-^,^{q,t) m,^^^{x,y) 

iJ^ri<X.,iJ, 

((-Pa,M7 PiJ.v))q,t « Sx^Sf^r, (1-15) 

where the ordering on pairs of partitions and the scalar product are respectively defined in (|1.14p and (jl.lip . 

If we let m and A'^ — m go to infinity, we obtain double Macdonald functions. We will nevertheless restrict ourselves 
to the finite case in this article. It should also be stressed that there is no solution to the two conditions (|1.15p in 
the non-stable sector if the ordering (jl.l4p is used. Therefore, in order to interpolate between the usual and double 
Macdonald polynomials (corresponding respectively to the cases m = and m > n), the construction relying on the 
super-dominance ordering (jl.Sp is necessary. 

1.2. Statement of the main results. As already pointed out, the bisymmetric version of the monomials and the 
power-sums display a very simple factorization pattern, irrespectively of the value of m. Such a generic factorization 
is not expected to be observed for the Macdonald bisymmetric polynomials. However, it turns out that in the stable 
sector, such a factorization occurs, albeit in a non-obvious way. Using the plethystic notation (reviewed in Section [5]), 
the factorization reads (cf . Theorem 2]) 

P^"*^*) [Y] , (1.16) 

where P'^'*'\x) denotes the usual Macdonald polynomial P\{x]q,t). 

We briefly digress in order to comment on the consequences of this result for the Macdonald polynomials with 
prescribed symmetry [T]. Let -Aj'^' be the i-antisymmetrization (or Hecke antisymmetrization) operator acting on the 
variables a;, S^^"^ be the t-symmetrization (or Hecke symmetrization) operator acting on the variables y, A((a;) be the 
t-Vandermonde determinant in the variables a;, and Ej(x,y; q,t) be the non-symmetric Macdonald polynomials in the 
variables xi, . . . , ccm, J/i, . . . , J/iv-m- It is known [7] that the double Macdonald polynomial (via the superpolynomial 
construction) P\^f^(x,y; q,t) is related to the Macdonald polynomial with prescribed symmetry A^i^^ s[^^ E^(x,y]q,t) 



P\,ti{x,y;q,t) = P, 



(q.qt) 



l-qt 
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through 

Px,^ix,y;q,t) cx -L^aI""^ S^"^ E^{x,y; q,t) (1.17) 

where 7 is any composition that rearranges to the partition (A + (5™) U fj,, and where oc means that the result holds up 
to a constant. The factorization (|1.16t means that if we take any non-symmetric Macdonald polynomials indexed by a 
composition of sufficiently low degreqj, i-antisymmetrize with respect to the first m variables, and t-symmetrize with 
respect to the remaining ones, then the result is either zero (if there are repeated entries in the first m entries of the 
composition) or, quite amazingly, a t-Vandermonde determinant times a product of two Macdonald polynomials! We 
should add that a degenerate case of ()1.16p was already known: dividing the expression in [1] Proposition 2] by the 
f-Vandermonde determinant yields (|1.16p for A = 0. 

The factorization ()1.16p offers the royal road to the study the properties of the double Macdonald polynomials. To 
a large extent, this study amounts to lift to the factorized form known properties of the usual Macdonald polynomials 
(cf. |19[ Chapter VI]). In this way, we readily obtain the norm, the duality and the evaluation (respectively given by 
Corollary [HI Proposition [7] and Corollary H]) . As a side result, we point out an interesting consequence of the duality: 
a simple identity involving the product of four Littlewood-Richardson coefficients (see Proposition [8|) whose building 
blocks are certain coefhcients studied in |8j. 

Implementing the plethystic substitutions X X and X + Y 1-^ {X + Y)/{1 — t) on the r.h.s. of (|1.16l) defines the 
modified double Macdonald polynomials: 

HxA^,y;q,t) ^ hI^-''^ [X + qY]Hl^'''^tX + Y] , (1.18) 

where h['''*\x] = J^''^*' [X/{1 - t)] is the modified Macdonald polynomial iJ^^'*\x) is the integral form of the Macdon- 
ald polynomial pI*'*-* (x)). The expansion of H\^f^{x, y; q, t) in terms of the Schur functions associated to the irreducible 
characters of i?„, namely s\^fj,{x, y) = s\{x) Sf^{y), define the P„ Kostka coefficients K,^^^ a,/j(9, t)- 

H\,^{x,y;q,t) = ^K,^^^ x,f,{q,t)Sf,^^{x,y). (1.19) 

We first justify the Bn terminology by observing that i^K,7 a,/j(1, 1) is equal to the dimension of the irreducible rep- 
resentation of Bn indexed by the pairs of partitions (k,7) (see Proposition 1101) . This is the first genuine contact with 
the hyperoctahedral group. We then show that the basic properties of the generic i?„ Kostka coefficients, namely, their 
positivity and symmetries, are immediate consequences of the factorization ()1.18|) and the related properties of the 
usual g,i-Kostkas (cf. Proposition [TT] and Corollary [13]) . 

Next, we define a deformation of the Nabla operator [4], denoted V^, whose eigenfunctions are H\^^{x^ y; q, t^^) and 
whose eigenvalues are given by a specific ratio of two P„ Kostkas. Somewhat surprisingly, we can evaluate the Schur 
expansion (which happens to be positive) of V^sg („) exactly (cf. Proposition I17p. From the ensuing expression, we 
deduce the following two results which will provide our most significant connection with the hyperoctahedral group 
(using the notation [n]q^t ~ (<?" ^ t")/{Q ^ t))- 



(V S0;(„), S0;(„))j 



(qtp) 



2n 
n 



(1.20) 

(V S0,(„),P0,(in))B = ^^^■^^n-l)/2 J ' (1-21) 

where (•, ■)b is the hyperoctahedral version of the Hall scalar product. 

In |15j . Haiman conjectured that for every Coxeter group W there exists a doubly graded quotient ring of the 
coinvariant ring whose Hilbert series Hilbg.t(P'^) satisfies 

m\ht-i^t{R^) = q;^ _^ _ (^ 22) 



More precisely, a low degree is relative to the values of m and N — m, and must be such that the stable-sector defining condition is 
satisfied: m, N — m > \ j\ — m{in — l)/2. 
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The existence of such modules has been demonstrated in [13j using the representation theory of Cherednik algebras. 
When W — Bn, this formula specializes to 

Hilb,-,,(i?-")^ri^V, (1.23) 



which is exactly the rhs of (|1.2ip when q = t ^. Furthermore, it is known that the alternating component of i?^" 
is given by the t-Catalan for the reflection group Bn 

which again corresponds to the rhs of (jl.20p specialized to q ^ t^^ . It is thus natural to surmise that 

Frobt-i,t(i?^") ^ Vf^,-iS0.(„) (1.25) 

where the symbol ~ means that the equality holds up to a relabeling of the indices (to ensure that the trivial module 
appears only at bidegree (0, 0), we probably need, in view of Corollary [201 to relabel the indexing pair of partitions as 

As for the generic q^t-case, it does not appear that Frob^.f (i?'^") ^ V'^S0:(„) (at least not with the natural grading 
stemming from [T?). Already in the n = 2 case, the _B2-Catalan is [5]q.f + qt[l\q^t (see for instance [12]) while (|1.20p 
gives ([5]g_t + q^t^[l]q^t) / qt (which is in some sense a homogenization of the i32-Catalan). 



1.3. Outline. Apart from the brief Section [2 reviewing the notation, and the Conclusion, the article is essentially 
divided in two parts. The first one. Section [31 is devoted to the study of the double Macdonald polynomials defined 
in Theorem [2] The pivotal result is the establishment of an equivalence between two scalar products, from which the 
factorization form (|1.16p is deduced. The rest of the section is concerned with the derivation of direct consequences of 
this main formula. The second main part. Section [31 is concerned with the investigation of generalizations of the Nabla 
operator. After reviewing how this operator is defined in the usual case, we present our heuristic approach that yields, 
among other things, the results mentioned in the previous subsection. In the short Conclusion, we reassert a new role 
for the Macdonald superpolynomials as the precise and fully explicit objects that interpolates between the Sn and i?„ 
versions of Macdonald polynomials. 

Four appendices complete this article. Appendix [X] contains the details of the statements made in the starting 
subsection concerning the transformation of the monomials and the power sums, from superspace to bisymmetric 
functions. Appendix [B] is mainly concerned with the proof of the equivalence between the two dominance orderings 
()1.8p and (|1.14p in the stable sector. This crucial result is relegated to an appendix because it is fairly technical and also 
because in disguised form, it is probably known. As discussed in the Conclusion, the results demonstrated here for the 
double Macdonald polynomials can be readily lifted to superspace. In Appendix [Cl we show that certain conjectured 
results in superspace are now validated in the stable sector, by matching the statements in |6l [7] with those of the 
present paper. Tables of Bn Kostka coefficients up to n = 3 are presented in Appendix IdI 

Acknowledgments. The authors are extremely grateful to Patrick Desrosiers and Stephen Griffeth for helpful discussions. 
This work was supported by NSERC, FQRNT, FONDECYT (Fondo Nacional de Desarrollo Cientffico y Tecnologico 
de Chile) grant #1090016, and by CONICYT (Comision Nacional de Investigacion Cientifica y Tecnologica de Chile) 
via the "proyecto anillo ACT56" . 



2. Definitions 

A partition A = (Ai,A2,...) of degree |A| — '}2n\i is a vector of non-negative integers such that Ai > A^+i for 
i — 1,2,.... The length ^(A) of A is the number of non-zero entries of A. Each partition A has an associated Ferrers 
diagram with A^ lattice squares in the i*'* row, from the top to bottom. Any lattice square in the Ferrers diagram is 
called a cell (or simply a square), where the cell is in the ith row and j'th column of the diagram. The conjugate 
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A' of a partition A is the partition whose diagram is obtained by reflecting the diagram of A about the main diagonal. 
Given a cell s = in A, we let 

ax{s)^Xi~j, and lx{s) = X'j - i . (2.1) 

The quantities ax{s) and l\{s) are respectively called the arm-length and leg-length. We will also need their co- version: 

a'xis)^j-l, and l'x{s) = i - 1 . (2.2) 

We say that the diagram fi is contained in A, denoted C A, if /i; < A; for all i. We also let X + ^ be the partitions 
whose entries are (A -I- fj,)i = Ai -I- /li, and A U /i be the partition obtained by reordering the entries of the concatenation 
of A and fi. The dominance ordering on partitions is such that A > /i iff |A| = and Ai -|- • • • -|- Ai > -I- • • • -|- for 
all i. 

The Macdonald polynomials Px{x; q, t), in the variables x — xi, X2, ■ ■ ■ , are characterized by the two conditions |19| 

1) P\{x] q, t) — m\ + lower terms, 

(2.3) 

2) (PA,PM>9,t=0 if A^Ai- 

The triangular decomposition refers to the dominance order on partitions and the m\s are the monomial symmetric 
functions: 

E'^^(i)---^^W (2-4) 

where the prime indicates a sum over distinct terms x^)^-^s^ ■ ■ ■ x'^^^^y The orthogonality relation is defined in the power- 
sum basis p\ =PAi • • -PAf , with pr = Y.i>i A > as 



^(^) I _ A. A(^) I _ A. 

(pA,PM)9.t = <^AM^A(g,0 where zx{q,t) = za H = H *"'^'^"^(*)' 11 I ' ^^'^^ 



n\{i) being the number of parts in A equal to i. We stress the notational distinction between the scalar product (•, ■)q^t 
and its bisymmetric version {{■,-))q^f 

The Jack polynomials P\{x;a) and Schur functions s\ can be defined respectively as the limits q = t",t —i' 1 and 
q ~ t oi the Macdonald polynomials. In the latter case, the scalar product (|2.5I) reduces to the Hall scalar product 
(PA,Pp) = Sx^zx which is such that (sa, s^) = ^^t^- 

We will use the language of A- rings (or plethysms) [3l [18]. The power-sum pi acts on the ring of rational functions 
in xi, . . . ,XN,q,t with coefficients in a field K (usually taken to be Q) as 



^2a '^a'Ua 
L/3 daVp 



V d 7/ ^ ' 

where c^, d/j S K and where Mq, vp are monomials in . . . , xn ^ q, t. Since the power-sums form a basis of the ring of 
symmetric functions, this action extends uniquely to an action of the ring of symmetric functions on the ring of rational 
functions in xi, . . . , xn , q,t with coefficients in K. In this notation, a symmetric function f{x) is equal to /[^], where 
X = {xi,X2, . ■ ■ ,xn) and X = xi+X2 + -- - + xn. 



3. HYPEROCTAHEDRAL MACDONALD POLYNOMIALS 

3.1. A remarkable factorization property. As indicated in the Introduction, we are interested in the stable bisym- 
metric version of the Macdonald polynomials as defined in Theorem [2] The stability property is captured by the 
condition m > n. In other words, if m and N — m are sufficiently large (> |A| + |/^|), then the bisymmetric Macdonald 
polynomial Px,p,{x, y; q, t) stabilizes, in the sense that its monomial expansion becomes independent of m and N . Within 
the stability sector, we can thus let m — > c» and N — m ^ oo, and obtain "double Macdonald functions" indexed by 
two infinite sets of indeterminates x = xi,a;2, . . . and y = yi,y2, ■ ■ ■ (corresponding respectively to xi,. . . ,Xm and 
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Xm+1, ■ ■ ■ ,xn in the limit 7n — > oo and N — m ^ oo). The distinction between "functions" and "polynomials" will not 
be necessary here and we shall always consider that our alphabets are finite. 

From now on, we will use the "plethystic" notation which is central to the derivation of our results (see Section[2]). For 
the remainder of this article, X and Y will stand respectively for X — xi+X2 + - ■ ■ + x„i and Y = yi+y2 + ■ ■ ■ + llN-m = 

Xm+l + • • • + XAT. 

We first prove that the scalar product (ll.lip can be rewritten in a much more convenient way for our purposes. It 
is in some sense the deepest result of this section. 

Lemma 3. If m > \X\ + and N — m > |A| + \fi\, then the scalar product is equal to the scalar product ((•, •))' 

defined as 

r +^ 1 r i 

Pu, [Y]))' = 5x„Si_,^q^^^ zx{q, qt)zi_,{qt,t) . 



'x+^-^Y 



1 - qt 



[Y] ,Pu 



l-qt 



(3.1) 



Proof. The scalar product (jl.lip can be rewritten as 

{{sx[X]p^[X + YIs4X]p4X + Y]))^^t = gl^l(sA, s.) {p^.,Pu.)g,t = q^^^Sx, x S^^z^iq,t). (3.2) 
Equation (|3.2|) is then equivalent to 

{{px [X]p^ [X + Y] ,p, [X]p^ [X + Y]))g^t = Sx,5^^q\^\zxz^{q,t) (3.3) 

given that the Hall scalar product is such that {px,Pu) = SxuZx- Notice that by homogeneity the extra factor g''*'! can 
be carried from p.2p to p.3p . 

We now show that the scalar products ((•, ■))q,t and ((•. •))' coincide. Let 



Pr 



X+^-^Y 



1-qt 

The quantities Ur and are related to Ur and Vr through 

"(l-g)X , q{l-t){X + Y) 



= Ur, Pr [Y] — Vr, Pr [X] = Ur, and Pr [X + Y] = Vr- 



l-qt 



1-qt 



l-qr _ q-{l-tn . 

-Ur H : Vr 



1 - 



1 - gT 



and 



Vr = Pr[—X + X + Y] = —Ur + Vr 
d 



With respect to the scalar product ((•, •))', we have (using dx — ^) 



= —q rdu 



1 - q^f' 



and 



1 - 

Vr = rOv 

1-V 



while with respect to the scalar product 



■) '//q,t, 



we have 



Ur — cfrdu^ and v'^ 



1 - f 



rdv^ 



(3.4) 

(3.5) 
(3.6) 

(3.7) 
(3.8) 



where and h-^ are defined respectively such that {{hf,g))q^t = {{f , h-^ g)} q.t and {{hf,g))' — {{f,h^ g))' . Given that 
{uxv^}x,fi is a basis of the space of bisymmetric functions of a given total degree n = \X\ + j/uj, the lemma will follow if 
we can show that 

{{uxv^,u„Vuj)y = {{uxV^,U^Vu,))q,t , (3.9) 

or equivalently, that 

((U0W0,M3^ V^ Ui,V^))' = {{uiliV(t„Uxvj^U^V^)) q.t ■ (3.10) 

In order to do so, it suffices to compare the recursions induced by Ur and Vr', repeated applications of the recursions 
will, by homogeneity, either lead to zero or 

((■ti0W0, U0t;0))q,t ((M0i;0, M0U0))' = 1 (3.11) 

and the result will follow. 

Note that the conditions m > |A| + \fi\ and N — m > |A| + ensure that 1*1,1*2, . . . and vi,V2, ■ ■ ■ can be considered 
independent (and similarly for ui,U2, ■ ■ ■ and vi,V2, ■ ■ ■)■ Observe also from (13. 5p and (j3.6p that du^ and dy^ commute 
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with Us and Vs if r ^ s. Now let A = (r*^) U A, ^ = (r™) U fi, v — (r^) U i) and w = (r") U w, where A, fi, and Co do not 
contain parts of size r. On the one hand, we have from (13. 7|) 



On the other hand, using (|3.5p . (13.81) . and the chain rule for derivatives, e.g. 



, 1 — r/'' 

(3.12) 
(3.13) 



dUr dUr dUr dUj. dVr ' 



we get 

= {{u';v'^U^Vjj^,u[vl':Ui,Vc))q,t 

1 — g'' 



1 - q^r 

1 - 

„ l-<^ 



1 — g'^i'^ 



1 — g'' 

q,t - ^ _ ^^^. q''rn{{ufr^v'l!'u-^Vfi,uivl!:^^U0V^))q^t 



1 — g'' 



-1 



1 - g''^ 1 - r 



(3.14) 



1 - q^V 

Comparing (I3.12p and p.l4p we see that the two Ur recursions coincide. 
We now do the same for the recursions. From (13.81) we find 



1-r 



while, using (13. 6p and (13. 8p . we get 

{{uxVf_,,UuV^))q,t = {{u';V^U'^Vf„U^^V^Ui>Vc,))q,t 



(3.15) 



1 — g'' 1 — t*" 1 — g^ 



1 - r 1 - gT 



1 — g'' 
1 — (fV 

- ^ _ rn{{u';vl!'-^u~^Vf„ ul.v'^-^upvco))q,t 



(3.16) 



The two Vr recursions are thus also seen to coincide, which completes the proof of the assertion that the scalar products 
((•, •))' and ((•, ■))q^t are equal. □ 



We now have all the tools to establish our key result; the double Macdonald polynomials have a totally unexpected, 
albeit rather non-trivial, decomposition into a product of two ordinary Macdonald polynomials. As mentioned in the 
introduction, the special case A = of the factorization is essentially contained in [T] . 



Theorem 4. IJm> |A| + |^| and - to > |A| + |^|, then 



Px,tJ.ix,y;q,t) ^ 



[q.qt) 



1 — qt 



(3.17) 



where P^'*'^ (x) stands for the usual Macdonald polynomial P\{x; g, t). 
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It should be commented that although the faetorization of Theorem [4] seems highly asymmetric in X and Y, the 
plcthystic substitution p.8ip will transform it into the much more symmetrical expression (I3.82|) . 



Proof. We need to show that the products of Macdonald polynomials appearing on the rhs of (|3.17p are unitriangular 
when expanded in the monomial basis and orthogonal with respect to the scalar product ((•, ■))q^t defined in (jl.lip . 



The products of Macdonald polynomials pI'''*-* 



l — qt 



piqt.t) (basically by definition) orthogonal with 



respect to the scalar product ((•, •))' defined in p.ip (notice that by homogeneity the extra factor g''*'! does not have any 
effect on the orthogonality). By Lemma [3l the orthogonality is then immediate. 



We now show the unitriangularity of P; 



(q.qt) 



X + 

l — qt 



\Y] in the monomial basis. By triangularity of the 



Macdonald polynomials, both in the Schur and monomial bases, we get 



l-qt 



l-qt 



UJ<.fJ, 



where * stands for some irrelevant coefficients (that we will keep denoting We have 

"9(1"*), 



X+^-^Y 
1 — qt 



E 

pC;>',7Cz-' 



[X]. 



l-qt 



-Y 



E 



.[X] [Y] 



(3.18) 



(3.19) 



p<Zv,(y:\cr\ + \p\ = \u\ 



Note that in the last equation, we only used \a\ — I7I and I7I + \p\ — \v\ (there is no triangularity when 
expanded in the S(j[y] basis). Moreover, it is an elementary fact that 

s„[Y]m^[Y]^m^+^[Y]+ ^ * [F ] . 

I3<a+u: 



l-qt 



Therefore 



P, 



(«,«*) 



pi,t,t)^Y]= E E E *^pmmp[Y] 



l-qt 

From \ > V and 3 p, we obtain 

Ai H + \i>vi^ Vvi> pi + ■ 

while from ^> uj, \p\ + \a\ = \v\ = |A| and uj + a > j3, we get 

|A| + /ii + IpI + |cr| + cji + ••• + cj, > IpI + cji + cTi + •••+ + fT; > IpI + A + •••+ A . 



i'<\,u}<p pCi/,o-:|(T| + |p| = |i/| IB<a+u 



Pi 



(3.20) 



(3.21) 



(3.22) 



which proves the triangularity. The unitriangularity is immediate from the unitriangularity of the Macdonald polyno- 
mials when expanded in the Schur or the monomial basis. □ 



For later references, we state explicitly the Jack limit. 
Corollary 5. If m > |A| + and N — m > |A| + then in the limit q — f^^t — >■ 1 we obtain 

1 



P..,(x,2/;a)=Pi"/("+^» 
where pI°'\x) stands for the usual Jack polynomial P\{x; a 



X 



a + 1 



-Y 



p(a+l) 



(3.23) 
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It is important to stress that in p.23p . the plethystic notation is such that pr acts on the ring of rational functions 
in the variables xi, . . . , Xm,yi, ■ ■ ■ , UN-m over the field Q(q!). Consequently, a is not affected by the plethys that is, 



X 



1 



a + 1 



-Y 



Pr[X] 



1 



a + l 



Pr[Y]. 



(3.24) 



3.2. Norm. Using Theorem |4] we obtain rather directly the expression for the norm of the double Macdonald polyno- 
mials. 



Corollary 6. The norm of the double Macdonald polynomial P\^fi{x,y; q,t) is: 

{{Px.Ax,y;q,t),PxJx,y;q,t)))g,t = q\^^bx{q,qt)-^b^{qt,t)-^ bx,^{q,t)-\ 

where 

^^(g'^)^li l-W.s)+i^K^) - 



(3.25) 
(3.26) 



Proof. Let Z = X + 4^^- In Lemma [3] we have shown that the scalar product is equivalent to the scalar 

product 

{{px[Z]pf,[Y],p4Z]p^[Y])y = Sxi,Sf,^q^^^zx{q,qt)Zf,{qt,t) :^ q^^^ {px,p^)q^gt X {p^,Pu)qt,t- 
Therefore, the factorized form of Px.i_i{x^ y; q, t) also implies a factorization of its scalar product: 

{{PxA^^ y; 9' i), -PA.p(a;, y; q, t))),,* = g'^' {P\{x; q, qt), Px{x; q, qO)g,<?t x (Pfj-i^; Qt, t),Pt^{x; qt, t))qt,t , 
and the result follows from the norm of the Macdonald polynomials [HI VI. 4. 11] 

{Px{x; q, t), Px{x; g, i)),,t - bx{q, t)-\ 



(3.27) 
(3.28) 

(3.29) 

□ 



If we define the normalized version of the double Macdonald polynomials as 

QxA^^ y; <?, t) = bx,t,{q, t)Px^^{x, y, q, t), 
where bx,fj.{q,t) = ((P\,^, Px,^i))^f was defined in (|3.25p . we obtain 

{{Q\.fj^{x,y;q,t),P^^^{x,y;q,t)))q^t = (^Ai/^^itu- 



(3.30) 



(3.31) 



3.3. Kernel. The form (|l.lip of the scalar product leads to a natural generalization of the Macdonald kernel. Let u 
(resp. v) be the union of the alphabets {xi,X2, ■ ■ ■) and (j/i, 1/2, ■ • • ) (resp. (zi, Z2, ■ ■ ■) and {wi,W2, ■■■))■ Defining 

{tUiVf,q)oc -TT 1 

q'^XkZi 



n:^Il{u,v;q,t)^ll 



(3.32) 



we obtain by standard manipulations that 



n=<^zp((?,t) ^Pf,{x,y)pf,{z,w)\ l^-^sxix)sx{z)[ ^^zx,^{q,t) '^px,f,{x,y)px,f,{z,w) 

I M J I A J \,fj. 



The duality (|3.31l) then implies that 



n = ^Px,,_,{x,y;q,t)Qx,f,{z,w;q,t). 

X,fi 



(3.33) 



(3.34) 



'^This is easily seen; 



Pr 



X 



a + l 



-Y 



■ lim Pr 

9=t° 



l-qt 



t-n 



lini Pr [X] + ' pr [Y] = Pr[X] + —-pr[Y] 



1 - g''t 



1 



a + l 
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3.4. Specializations. In [6j, a picture describing the various specializations of the Macdonald polynomials in super- 
space was presented. Figure [2] gives the corresponding picture in the case of the double Macdonald polynomials. In 
the figure, there are two Hall-Littlewood limits {Px^^{x,y;t) and P\^^{x,y\t)), one Jack limit {P^^{x,y)) and a one- 
parameter Schur limit {s\,^{x,y;t) = P\^f^{x,y; q,t)) that specializes to the corresponding limits of the Hall-Littlewood 
and Jack limits. Given the factorized form p.l7p of P\^f_i{x, y; q, t), we can give each of these limits explicitly. The Jack 

Figure 2. Limiting cases of the double Macdonald polynomials 



P\,tL{x,y\q,t) 




s\.f,{x,y;t) 



limit was presented in p.23p . The Hall-Littlewood limits are 

Px^^ix,y;t) ^ .sxix)P^{y;t) and PxA^,y;t) = .sx[X + {1 - l/t)Y] P^{y;l/t) , (3.35) 

where Pfj.{y;t) is the Hall-Littlewood polynomial (the limit g = of the corresponding Macdonald polynomial). Finally, 
the Schur limit is 

Pf''^ [Y] , (3.36) 
which specializes to 

and sx,^ix,y) = sx[X + Y]s^{y), (3.37) 

where we recall that P*^"-'(j/) is the Jack polynomial. Not considered in Figure [5] are the limits q = I and t — I, which 
give respectively 

ex,f,{x,y) = ex[X + Y]s^{y) and mx.j^ix,y) ^ sxix)m^{y) (3.38) 
the analogs of the elementary and monomial symmetric functions. 

3.5. Duality. Let uix be the standard involution 

LOxPriX] = (-!)'■- Vm = {-IYpA-X] (3.39) 

which is such that wxSa[^] ~ sx'[X]. The involution ui — ujx ° ujy is such that for any elements a{q,t) and b{q,t) of 
Q(q, t) we have 

ijprHq,t)X + biq,t)Y]=a{q\nuJxPr[X]+b{q'',r)ujYPr[Y] = {-ly^Pr [a{q,t)X + b{q,t)Y] . (3.40) 
Hence uj acts as the usual involution on symmetric functions in any alphabet made out of a combination of X and Y. 



sx,t,{x,y;t) = P^ 



X 



-Y 



sx,p.ix,y) = sxix)s^{y) 



sr,{x,y) = P, 



(1/2) 



X + -Y 
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Let us now define the more general automorphism cj^^ as 



Pr[X + Y] = {-lY'^—i^p,[X + Y]= up. 



t fl-q 



q \l-t 



(X + Y) 



Ujf,tPr[X] = {-iyepr[X]^Upr[-tX] 



Clearly, the inverse of ujg^ is given by 



'-'v-\q- 



Proposition 7. The following dualities hold: 



where |A| + = n. In particular (see (|3.3ip ); 

((^5 {x, y, q, t), Pu,Lo{x^ t^^,q^^)))t-Kq-^ = ^^^5^, 



(3.41) 
(3.42) 

(3.43) 

(3.44) 
(3.45) 

(3.46) 



Observe that as expected, the i3„-analogue of the conjugation sends the pair of partitions ji, A to the pair of partitions 

Proof. We will prove (j?^ . Relation ((5^ will then imply ((H^ . 

First, from the factorized form of the double Macdonald polynomial ()3.17p . we have: 



[t-\{qt)-^) 



X 



{q-'){l-t-^)Y 



1 - [qt)-^ 



X 



(l-q)Y 



1 - qt 



p[f'''\Y] 



(3.47) 



where, in the second equality, we used the symmetry P^ 
explicitly. We have 



\X] = Pf'*'[X]. The duality can now be computed 



(q.qt) 



1-9 
t- -Y 

1 - qt 



^piqt.t) 



t\ 1-qt 



qj 1-t 



X 



^t\^\by{qt,q)Plf''^\Y](-] b {t , qt) P^l 



{t,qt) 



X 



Y{l-q) 
1 - qt 

(i-g)y 



1-qt 



where we have used p[''*^[tZ] = rl'^lp['^'*-'[Z] (r stands for any monomial in q and t) and the usual duality 



(9,t) 



l~t' 



= by{t,q)Pl'A\Z] 



Using the relation 6a (9, = (V'Z)''^'^A (9 ^,t ^), wc then obtain 



^^,PxA^,y;q,t)^t\^\b^,it-\{qt)-')byiiqt)-\q-')P^.,yix,y;t-\q-'), 



which completes the proof. 



(3.48) 
(3.49) 

(3.50) 



(3.51) 

□ 



The scalar product (|l.lip does not behave well in the limits q = t = and q = t = 00. An interesting consequence of 
Proposition 13 . 5 1 is that it can be used to relate the corresponding specializations of the double Macdonald polynomials. 
In effect, from the definition of w^j, we have that p.46p is equivalent to 



The limit q = t = 00 oi this result is then well defined and reads 

{{^1,1 Sti'[X + Y]sx'{y),Si,{x)su,{y)))i.i = Sxu^^, 



(3.52) 
(3.53) 
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The following identity involving products of four Littlewood- Richardson coefficients (which to the best of our knowledge 
is not in the literature) follows from the previous duality. 



Proposition 8. We have 



(3.54) 



^.7].<7,T 



where ^,r],a,T run over all partitions and cy^^ is a Littlewood- Richards on coefficient. Equivalently, if as in [8j we let 



the identity reads 



ri,T 



Proof. The scalar product (jl.lip in the limit g = i = 1 is such that 

Moreover, the action of lxj^^ on s^[X] sx[X + Y] is easily seen to be 

c^fi s^[X] sy^[X + Y] = {-\)\''\s^[X]sy[X + Y]. 
It will thus prove convenient to expand s^;[X + Y]sxi{y) and Si,{x)s^^{y) in the s^[X] sx[X + Y] basis. We have 
sAY] - sy[-X + X + Y] = ^4s^[-X].s,[X + Y] = 5]4(-l)l'^lsy + Y] . 



1,V 



Hence 



Similarly, after some straightforward computations (using again (|3.59p ). we obtain 

Su[X]s^[Y] = Y,{-\)\^\cl,,c-,,s^[X]s.[X + Y]. 

Finally, replacing the expansions (I3.60p and ()3.6ip in f|3.53p . we get thanks to ()3.57|) the identity 
The proposition then follows since 



where we used cJ^,,, — c\,> and — c^, 



7,?7,(T,r 



(3.55) 
(3.56) 

(3.57) 
(3.58) 
(3.59) 

(3.60) 

(3.61) 

(3.62) 

(3.63) 
□ 



3.6. Evaluation. We now provide an explicit formula for the evaluation of the double Macdonald polynomials. The 
first point to clarify is the way we could specialize the variables x and y. For this we recall that the most general 
evaluation of the usual Macdonald polynomial is fl9^, eqs VI (6.16)-(6.17)] 



P. 



1-u 
1 - t 



ll i_ga(.)^Us)+i -^^A(z^;g,0 



sex 



where the plethysm is such that: 



1 - u' 

1 - t 



(3.64) 
(3.65) 
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Considering the factorized expression (|1.16p , one sees that in order to evaluate the first term P, 
need to bring the argument in the proper form, that is, set X and Y such that 



(9,9*) 
A 



we 



X 



-Y 



(3.66) 



1-qt 1-qt 

This clearly requires X and Y to be of the form 

X = q''{l + qt+--- + {qtr) and Y ^ t\l + 1 + ■ ■ ■ + 1^-""), (3.67) 

and the condition p.66p further imposes a = 1 — m and h = m (so that the resulting u is q^'t^). We thus define the 
evaluation as 

^N,-m{P\.A^,y\q,t)) P\.f,{x,y;q,t) 



Corollary 9. With the evaluation defined in p.68|) and w\{u;q,t) defined in p.64p . we have 



Proof. The specialization gives 

X = Xi + X2 + ■ 

and 

Y = yi 



q-m—l qin — 2 



1 - qt 



J/Ar-m — t + . . . + t — t 

The evaluation of the two terms in the factorization (jl.l6p is immediate: 



1-t 



N-7n 



1 - t 



E 



N' 



P, 



{q,qt) 



X+^-^Y 



1 - qt 



1 p(9,9*) 
,(m-l)|Ar A 



and 



1 - 1 



N-ml 



1 -t 



^m\fi\ p{qt,t] 



1-qt 

^ _ -^N — rn ' 



1 - 1 



f"l^lw;^(t^-™;gt,i). 



whose product yields the announced result. 



3.7. Kostka coefficients. Recall that the integral form of the Macdonald polynomials is [191 VI. 8. 3] 

Jx{x;q,t) = cx{q,t)Px{x;q,t) 

where 

cxiq,t)^llil-q'^'-''h'(^^+'). 
sSA 

We define the integral form of the double Macdonald polynomials to be 

J\A^' 2/; <?' = cx{q, qt) Cf,{qt, t) Px^^{x, y; q, t) . 
Recall also the definition of the modified Macdonald polynomials 



X 

1 - t 



Hx{x-q,t)^j{ 

which, when expanded in the Schur basis, are such that (this is equivalent to [El VI. 8. 11] 

Hx{x;q,t) = ^Kf,x{q,t) s^(x) 



(3.68) 

(3.69) 

(3.70) 
(3.71) 

(3.72) 

(3.73) 
□ 

(3.74) 
(3.75) 

(3.76) 

(3.77) 

(3.78) 



where K^x{q^t) is the g,t-Kostka coefficient. It has been shown in [16] that Kfj_\{q,t) e N[9,i]. Recall also that 
Kfj,\{l,l) — X('i„-), where xfi") is value of the irreducible S'„-character at the class of the identity (HI VI. 8. 16]. 
Equivalently, ^^^(l, 1) corresponds to the number of standard tableaux of shape fi (cf. 3, eqs (2.1) and (9.7)]). 
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By analogy, we define 

Hx.j,{x,y;q,t) = if(Jx,^,{x,y;q,t)) 
where tp is the the homomorphism whose action on the power sums is 

Pn [X] ^ Pn [X] and pn [X + Y] ^ T^P" + ^] 

Note that the homomorphism tp is equivalent to the plethystic substitution 

1 



X ^ X and X + Y ^ 



(X + Y) 



(3.79) 
(3.80) 

(3.81) 



^From Theorem |4l after some straightforward manipulations, we obtain that 



(q.qt) 



X 



qY 



1-qt 1-qt 



jiqt.t) 



Y tX 



l-t l-t 



= iJ^"'"*^ [X + qY] H^"*'*') [tX + Y] 



(3.82) 



Our interest is to introduce Bn analogues of the Kostka coefficients by expanding Hx^^{x,y\q,t) in terms of the Schur 
functions associated to the irreducible characters of _B„ (see (THl p. 178]): 

sx,^,{x, y) = sa(x) s^(?/). (3.83) 

Observe that from Theorem [H these Schur functions correspond to the specialization q = t = Qoi P\^^(x] q,t), namely 
sx^f,{x,y) = Px^f,{x,y;0,0) (see also [6j[7]). 

Now define the Kostka coefficients Kf^^y x.p.{q,t) through the expansion 



HxA^^y^l^^^) = ^K^^y x,f,{q,t)s^,^y{x,y). 



(3.84) 



The reader is referred to Appendix [Dl for tables of Kostka coefficients up to degree n = 3. We ffist connect Ki^^j A.^t(l, 1) 
to the representation theory of the hyperoctahedral group i?„. 

Proposition 10. Let A and fi be such that |A| + \fi\ = n. Then ^^,7 A,/i(l, 1) is the dimension of the irreducible 
representation of Bn indexed by the pairs of partitions k,j. In particular, K^^^ x,p,{^, 1) does not depend on A and pi. 



Proof. When q = i = 1, it is known pf, eq. (9.6)] that 

i/A(x;l,l)-pl^l. 

It thus follows from ((XSa that 

HxA^^ y; i> 1) - H^x^^\x + F]^!'!) [x + Y] = {p,[x + r])i^i+i''i = \x + y] 

But it is known (see |19[ p. 178]) that 

Pp [X + Y]p, [X-Y]^Y. Xp.- y) 



(3.85) 
(3.86) 
(3.87) 



where Xp stands for the i3„-character indexed by the irreducible representation k, 7 at the class indexed by p, a. If 
we set p = 1" and a = 0, we obtain that 



Hx,f^{x, y\ 1, 1) pi" [X + Y]=^ Xir. J s^,y{x, y) 



(3.88) 



which proves the proposition since the class indexed by 1",0 is the class of the identity, in which case the character 
yields the dimension of the representation. □ 
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Proposition llOl implies that K^.,-^ A,/i(l, 1) is the number of pahs of standard Young tableaux of respective shapes k and 
7 filled (without repetitions) with the numbers {1, 2, 3, ... , n}, where n = |k| + |7|. For instance, i^(2,i),(i) A,/i(l, 1) = 8 
is the number of pairs of standard Young tableaux of shape (2, 1), (1): 



1 


2 


3 





1 


3 


2 





1 


2 


4 





1 


4 


2 





1 


3 


4 





1 


4 


3 





3] \T\ 2_J\ \T\ 



(3.89) 



We now show the positivity of the coefficients K^^j x^^{q,t). The proposition implies that the _B„-analog of the 
g,t-Kostka can be obtained from the usual g,t-Kostka (of S'„-type). Therefore, from the combinatorial point of view, 
this new hyperoctahedral perspective does not shed any light on the usual q, i-Kostka. In Section [5l we will discuss how 
an interesting new combinatorics appears in the non-stable sector that interpolates between the Sn and Bn cases. 



Proposition 11. We have 

where c^^ is the corresponding Littlewood- Richards on coefficient. In particular, Kf^^^ \^^{q^t) G N[q,t]. 

Proof. From ([5T7|) and we have 

Hx,^,{x, y; <?, = ^ -^-^(«' ^^M^ s,[X + qY] s^[tX + Y] 

U,UJ 

= E K,x{q,qt)K^M^t)clpc';^s^[X]sp[qY]sp[tX]s,[Y] 

= E K,x{q,qt)K^^{qt,t)c''^pc'^p,q^''h^P^s^[X]sp[X]sp[Y]s^[Y] 
E if.A(g,<zt)if^M(<zi,i)c:i;3C^,cSpC;^,gl'^ltl''ls«[X]s,[y] 

f ,cj,a,/3,p,(7,K.,7 

= E E ^) <^ c 2^)- 



(3.90) 



(3.91) 
(3.92) 
(3.93) 
(3.94) 

(3.95) 



This proves the first assertion. The positivity of ^^^,7 A,/i('?, then follows from the positivity of the (j,t-Kostkas and 



the Littlewood-Richardson coeflacients. 



□ 



Remark 12. Propositions [TOl and fTD suggest that there exists a bigraded module of the regular representation of the 
hyperoctahedral group Bn whose Frobenius series corresponds to the Schur expansion of H\^^{x, y; q, t). This point will 
be discussed further in Remark [22l 

Corollary 13. The Bn-analogues of the q,t-Kostka coefficients have the following symmetries: 

i^K,7A,p(g,<) =g"^''''^'^i"^^'^^i^y-'A,M('Z"\^"') and i^,,^A,p(g,i) = ^7',K'P',A'(i,9), (3-96) 
where n(X, /i) = n{X) + + n{p') + ri(/i). 

Proof. For the usual Kostka coefficients, the analogous form of the first symmetry is equivalent to (see e.g., 0] eq. 
(9.9)]) 

Hxix; q, t) = <z"(^') r(^) ujHxix; q-\t-^) (3.97) 
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where w is such that ws^ — s^/ . If we extend uj to the space of bisymmctric functions as ws^ [X] — s^' [X] and 
a;s^[y] — Sfj_>[Y] (which implies for instance that ijJSx[X + qY] = sx'[X + qY^, we obtain from p.82p 

HxA^,y-q,t) = [h[^" ''^'''^'"\x + qY]) {qty^^'h^^'^^u; (iJ^^*)"''*"') [fX + y]) 

= (^H['''''^'^'^''\q-^X + Y]) {qty^i^'h'^^^'h^^'^uj [X + t'^Y]) 

= g«(M'.A')in(A,M)^ + LO (i/««*)"-*")[X + t-ly]) (3.98) 

= g'-'(A''-A')fHA,M)^^^_^(y^2,.^-l^i-l) (3.99) 

But this amounts to 

HxAx,y;q,t) = q^^^^' ■■^'h^^^-'''> ujBHx,^ix,y; q-\t-') (3.100) 
where uJBSx^^_i{x,y) — s^/.a' (x, y), which implies the first relation. 

For the second relation, we use the other Macdonald symmetry |3i eq. (9.8)] 

Hx{x;q,t)^LjHx'{x-t,q) (3.101) 

to obtain 

Hx,^{x, y; q,t) = lj (^f '^^ [X + qY]) u [tX + Y]) = ujH^,^x' (y, x; t, q) . (3.102) 
But this is equivalent to 

Hx,,i{x,y;q,t) = ubH^^ ^x'{x,y;t,q) (3.103) 

and the result follows. □ 

It is known that for |A| = n [11, Propo. 3.5.20] or [T9l, ex. 2 p. 362] 

K^,,)x{q,t)^t<^^ and K^,,.)x{q,t) = q<^'^ (3.104) 

These correspond to the two inequivalent representations of dimension 1 of Sn ■ In the case of _B„ , there are 4 inequivalent 
representations of dimension 1. The corresponding Kostkas, which will prove important in the next section, are given 
in the next Corollary. 

Corollary 14. Let |A| + |/^| = n. We have 

Kin)fi A,m(9, t) = qnWtM+nW+ni^) ^^^^^^^ ^^^(^^ ^) ^ ^|A|+n(A)^„(A)+„(^) (g^^Qg) 

By symmetry, i.e. using the first relation of (|3.96[) . we also have 

^0,(1")a.m('?>*)=9'"^"^^'^^"^'''^^"^'''^ and a.^C?, = 9"^''^+"^^'^^'^'+"^^'^- (3-106) 

Proof. We will only prove the expression for ii'0,(„) x,fiiQi t) since that of K(^n).^ x.fiili t) can be obtained in a similar way. 
From (|3.82|) . -K^0 („) ;^^(g, i) is equal to the coefficient of s%^(n){x,y) — S(„)[y] in Hx,^i{x,y;q,t). In the product form 
(P:^ we can thus set X = and search for the coefficient of S(„)[y] in iJ^'^'''*^ [gy]H^'''*^ [F] = gl^lH^'^'*'[y]i7^'^*^*'[r]. 
Let us first expand the latter product: 

= Y,K^^{^q^qt)Kp^{qt,t)s^[Y]sp[Y] = ^ K^x{q, qt) Kpp{qt,t) c^^^ s^[Y]. (3.107) 

For ijj = (n), it follows that the non-vanishing Littlewood-Richardson coefficients c^p are equal to 5r^^[ni)^p,(n2) fo'^ 
non-negative integers ni and satisfying ni + n2 = n. The homogeneity of the Kostka coefficients then imposes 
ni = |A| and 712 — l/^j. The desired coefficient is thus equal to g'^'i^(„i) a('7j 'Z*)-f^(n2) Using (|3.104p . the result 

follows. □ 



This is seen from 

a;.;,[X + gy] =^c^,s^[X]glH.,[y] =^c;;,,s^4X]gl''l.,y[y] =^c;;;^,Vm«.'k>'] = ^A'[^ + '7>'] 
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4. The Nabla Operator 

4.1. Review of the usual case: Nabla operator and Frobenius series. A useful technique to demonstrate that 
a given symmetric polynomial is Schur positive is to link this expansion to the decomposition of a representation into 
irreducible ones. By means of the characteristic map, a symmetric polynomial is transformed into a class function of S'„. 
Under this map, a Schur function is associated to an irreducible character of S'„. It follows that a symmetric function 
F associated to a representation is Schur positive: this symmetric function can be expanded as -F = ^ cxs\, where by 
construction c\ is a nonnegative integer since it corresponds to the multiplicity of the irreducible representation A in 
the larger representation under consideration. 

Let 

Hxix; q, t) = r^^^Hxix; q, i"') = ^ K^.x{q, t)s^ix) , (4.1) 

(hence K^\[q^t) = t^'^'^^ K^\{q,t~^)). Garsia and Haiman [12] were able to construct bigraded S'n-modules H.^ (the 
so-called Garsia-Haiman modules) such that 

Yio\,t{'H^,)^Hx{x-q,t). (4.2) 

(More precisely, this result is conjectured in [1^ and proved in [16].) The Garsia-Haiman modules belong to the larger 
module Vn of diagonal harmonics whose dimension is [n + 1)""-'^. Quite remarkably, the bigraded Hilbert series of I?„ 
can be calculated from the the operator V introduced by Bergeron and Garsia ^ (see also [5] and P], beginning of 
Section 9.6) and defined as follows: 

VHx = q^'^^'h^'^^^Hx. (4.3) 

In other words, V is defined from its action on Hx and because these polynomials form a basis, this provides an action 
on any symmetric function. It has been proven (and this is a highly non-trivial result) that the action of V on e„ , when 
expanded in the Schur basis, 

Ve„ = ^CA(g,i)sA , (4.4) 

A 

can be interpreted as the bigraded Frobenius series of 2?„ (see e.g., [TT] Theorem 4.2.5] and Sections 3.5, 4.1 and 4.2 
therein for the subsequent results of this section). Moreover, in the case q = 1/t, it has been shown that 

(V,=v,e„,p?)=(Mi)" \ (4.5) 

where 

ik]t = j^ = {l + t+... + f'-'). (4.6) 

Specializing this result to t = 1 one recovers {n + 1)"^-'^ as the dimension of 2?„. Note that for q and t generic, there is 
no factorization of the type 

(Ve„,pr) = (/(g,0)""\ (4.7) 
for some polynomial f{q,t) G N[(7,t]. The coefficient of e„ in Ve„ 

C„(g,t) = (Ve„,e„), (4.8) 

is also of particular interest. It corresponds to the bigraded Hilbert series of the subspace An of alternants in 
There is a combinatorial interpretation for C'n{q,t) (reviewed in jl4l Chapter 3]), but there is no known closed- form 
expression. However, when q — 1/t, Cn{q,t) reduces to 

' 2n 



which is a t-analogue of the Catalan number 



n 



(4.9) 



a^^r )■ (4.10) 



n + l\n 

The dimension of the subspace of alternants An is thus given by the n-th Catalan number 
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4.2. Heuristic strategy for the Nabla operator in the hyperoctahedral case. Proposition [10] provides a con- 
nection between the hyperoctahedral group and the double Macdonald polynomials. In this subsection, we will deepen 
this connection by the introduction of Nabla operators that will produce another i?„ datum, namely the Bn version of 
the dimension formula (n + 1)""^. 

Recall that the group Sn is a Coxeter group of type A, namely The formula {n + 1)"^^ turns out to be 

the An-i version of the dimension formula (h + ly [13j, where h and r are respectively the Coxeter number and the 
rank of the corresponding Coxeter group (h ~ n and r — n ~ 1 for An-i). Now the hyperoctahedral group of rank 
n is equivalent to the group of signed permutations of n objects. Its generators are {sq, si, s„-i}, where so is the 
sign change, and where the remaining generators are the elementary transpositions that generate 5„. This description 
makes clear that the hyperoctahedral group is the Coxeter group B„. Given that for _B„ the Coxeter number and the 
rank are respectively h = 2n and r = n, the expected i?„-form of the dimension is {2n + 1)". This is precisely the value 
we will obtain. 

We first need to formulate the correct definition of the Nabla operators in the _B„-case. In order to do so, let us 
briefly reexamine the usual case from an heuristic-constructive point of view. From p.l04p . the eigenvalue of V on Hx 
corresponds to K;^in'fx{q,t~^)/ K(^n)\{'lit~^), that is, to the ratio of the Kostka coefficients associated to representations 
of Sn of dimension 1. This is the guiding observation that we will use to define the B version of the Nabla operator. 

As noted before Corollary [TH in our case there are four pairs of partitions whose Kostka coefficient is equal to 1 
when q — t ~ 1: 

Kin)3 A,m(1. 1) = ^(1"),0 A,m(1. 1) = ^0,(n) A,p(l, 1) = ^^0,(1") A,p(l, 1) = 1- (4.11) 

There are thus four choices for the rescaling factor that sets equal to 1 a given g, i-Kostka. We will choose to rescale 
^(i").0 x,fiiQj to 1 by redefining H\^f^{x, y; q, t) as (this choice has no impact on the definition of a Nabla operator) 

H\,^{x, y; q, t) = — Hx,t,{x, y; q, t^^) = V A'^ ^,^((7, t)s^.y{x, y). (4.12) 

^(i"),0 a,m(9:^ ) 

By (I3.106|) we have the more explicit relationship between Hx^^ and Hx^f^'. 

HxA^, y; q, t) - <z-"(^')-"(''')il^l+"(''')iJA,M(:^, y, q, t-') . (4.13) 
Observe also that Ki^^^ X4i{q,t) now belongs to N[q^^,i^^]. 

Following our guiding observation and (14. lip , there are two natural and independent ways to define a Nabla operator 
in the B case, that is, two natural choices of the eigenvalue assigned to Hx,fj,{x, y; q, t). First, we can let the eigenvalue 
be 

i^(l"),0A,^(g,i-l) . ^. ) 

Note that the two pairs of partitions, (0, (n)) and ((1"), 0), are conjugate of each other. Hence the first Nabla operator 
is defined as 

V^i?A./x = gl^l+"(p',A')^|p|+ft(A,M)^^_^ ^ (4,15) 

where 

n{X,n)=n{n')-n{X)-n{fi). (4.16) 
Alternatively, we can also define the eigenvalue of the Nabla operator to be the ratio: 

-^0,(1") A,plg,t ) ^ g' ' 't ^ |A|-n(A)+n(A')+n(M')Jphn(M')+"(A)+n(M) ^ 1 7^ 

K(^n)fixAl,t-') g"(A)t-|H-n(A)-n(M) « ' ^^'^'^ 

which leads to 

The operator does not appear to have generic noteworthy properties. In particular, it is not always Schur positive 
(up to an overall sign) when acting on the Schur functions s^,,^ (not even when acting on S0;ii>)- However, there seems 
to be a surprising connection with the (7,i-Catalan Cn+i{q,t) (as if i?„ knew something about 5„+i!) 

Conjecture 15. The coefficient o/s0;in in V^S0;in is equal to Cn+i{q,t) . 
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/From these two commuting operators, one can consider their product V V , whose eigenvalue is readily evaluated 
from (I4.15P and (I4.18|) to be g2|A|^2|^|^ rpj^^ factor 2 in both exponents further indicates that the square root of V^V^ 
is well-defined and thus more fundamental. This operator, defined as 



(4.19) 



actually appears to be more interesting. Indeed, its action on a _B-type Schur displays a Schur-positive expansion (up 
to an overall sign). 

Conjecture 16. The Schur- expansion coefficients of sx^n belong (up to an overall sign) to N[q^^,t^^] for 

every integer i and every pair of partitions A, fj,. 

In the remainder of the section, we will be solely concerned with the operator which connects with (2n + 1)". 

4.3. The explicit action of on („). The analog of Ve„ = Vs^in) in the i?„ case appears to be V'^S0 („), with 
defined by (j4.15p . The next proposition gives a simple expression for V'^S0 



Proposition 17. We have 



where 



(gt)(S) 



[n]g,tX + [n + l]g^tY 



q-t 



(4.20) 
(4.21) 

(4.22) 



Proof. We recall the simple identity (see [19] and e.g. [Ml eq. (1.61)]) 

n n 

hn[Z + W]=Y, K,-f\Z] h,[W] = ^(-l)"-^e„_,[-Z] h,[W] , 
where the second equality follows by the duality relation: with uj such that 

Upr[X] = i-iy-'priX] = {-lYpr[~X] , 

it follows that 

Px[~X] = {~l)^^^wpx[X] =^ hx[-X] - {^l)^^^uhx[X] = (-l)l^leA[X] 

Using he\tZ] = t^hi\Z] (and similar results when q is replaced by t or kg is replaced by e^), and suitable choices for Z 
and W, we can write 

n-e. 



(4.23) 
(4.24) 



f=0 



X + qY 
l-q/t 



hf 



X/t + Y 



l-q/t 



-X/t - qY/t X/t + Y 



l-q/t 



l-q/t 



hn[Y]. 



Since S0,(n) (2;, J/) — S(n)(y) — hn[Y], we have thus obtained the following expansion 

■S0,(n) = 



f ly-' 


'X^qY' 


hi 


'X/t + Y' 


\—t) 


i 1 - 9/^ J 




i 1 - 



(4.25) 



(4.26) 



To obtain the action of on S0.(„), it thus suffices to find its action on each term of the sum on the rhs. Recall the 
following expressions (see [T^ eq. VI. 4. 8 and 4.9) 



and 



Pllf[Z]^h, 



[l-t) 



.(1-9) 

where the symbol oc means that the result holds up to a multiplicative constant. Therefore 



H, 



(q.q/t) 



[X + qY] cx ek 



X + qY 
1-q/t^ 



and 



ijj/*^^/*) [X/t + Y]ochk 



X/t + Y 
[l-q/t ^ 



(4.27) 



(4.28) 
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which implies, using p.82p and (|4.12|) . that 

\X + qY] \X/t + Y] 

H{i^-i.).(t){x,y;q,t) c^Cn-i ^ _ hi ^ _ . (4.29) 

In other words, the product Cn^ehi appearing in the decomposition of S0.(„) in (j4.26p is proportional to H^in-t'^ i^i-^, 
which is itself an eigenfunction of V'^. Its eigenvalue is read off (|4.15p specialized to the case A — (1"^^) and fi — {£): 

, (]sn(n+l)/2-nl 

V^ff(i.-.),(,) = (1) . (4.30) 
After straightforward manipulations, we thus get 



n(n+l)/2 



ti'- 



1=0 



^n+l 



(-l)"-^e„_, 



X + qY 
1 - g/t J 



hi 



X/t + Y 



l-q/t 



^^n(ri+l)/2 " 



-X-qY 



he 



qt 



ri(n-l)/2 



hn 



l-q/t 
[n]g,tX + [n + l],^tY 



l-q/t 



(4.31) 
□ 



In general the action of V'^ on Schur functions, contrary to that of V, is not positive (up to an overall sign) when 
expanded in the Schur basis. But what is remarkable here is that the Schur expansion of the action of on S0.(„) can 
be given in a closed form for q and t generic (while a similar result for Ve„ can only be established when q = 1/t). 



Corollary 18. The action ofW^ on S0:(„) expanded in the Schur basis s\^^ — s\{x)Sfj_{y) is given by 



(4.32) 



(qt)'^'' X.J. 

In particular, the Schur expansion coefficients belong to N[q'^^,t^^]. 

Proof. Using and the Cauchy identity (see JS] and e.g. [H eq. (1.63)]) /i„[Xy] = J2f,\-n «m[-'^]spM> we find 



n 

i]g^tX + [n + l]g^tY^ ^^hn-e[[n]g^tX he [n + 1]^,*^ 



n 



1=0 Xhn-e 



tJ.\-i 



(4.33) 



The corollary then follows from Proposition[T71 The positivity is immediate since [n]q^t € ^[q, t] and the Schur functions 
are monomial positive. □ 

e Schur functions are orthonormal 

(4.34) 
(4.35) 



The Bn analog of the Schur functions are orthonormal 

with respect to the i3„ analog of the Hall scalar product defined afl 

{px[X + Y]p^[X - YIp^[X + Y]p,[X - Y])b - 5xu5^,Z)^z^ 2^(a)+^(a.) 



''The scalar product is given in 1191 Appendix B]. To be more specific, it is the specialization of (5.3') therein to the case Bn = C2 ^ Sn 
(discussed in more details in the example on page 178). The power of 2 is due to the fact that the centralizer of each element in C2 is of 
order 2. 
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We now extract from the previous Corollary a closed-form expression for the Bn analog of the g,t-Catalan (V^e„,e„) 
(compare for instance with (|4.9p in the case q — 1/t). As expected, it reduces to (^^) when q = t = 1. 

Corollary 19. 



(V^S0;(„), S0.(„))b — — Y 

{qtp) 



2n 

n 



q,t 



where 



In particular 



2n 
n 



[2n],,t! 



and Nq,t! = [n],,* • • • [2],,t [l]q^t ■ 



(Vf=i/4 S0;(n),S0;(«)>S = J, 



2n 

n 



(4.36) 
(4.37) 
(4.38) 



Proof. Given the orthonormality (|4.34p . to evaluate (V^S0;(„), S(i)-i^n))B it suffices to compute the coefficient of S0^(„) in 
V^S0_(„). This coefficient corresponds to specifying A = and /i = (n) in the rhs of (I4.32p : 



'^0;(n) ; ^0;(n 



^77^ lT-n[[n + l]q,t] = 



The previous equality follows from [n + l]q^t ~ q"'[n + l]f/g, the relation 



2n 
n 



q,t 



' 2n ' 




' 2n ' 








n 


n 









t/q 



and the identity jT9] 



hn[[n + l]t] 



2n 
n 



Corollary 20. We have 



(V-^S0.(„), S(i«).0)b = 1 



(4.39) 

(4.40) 

(4.41) 

□ 

(4.42) 



Proof. As in the proof of the previous corollary, it suffices to compute the coefficient of S(i").0 in V^S0^(„), which 
corresponds to specifying A — (1") and /i = in the rhs of (|4.32p : 



(V^S0.(„),S(i 



(n), 5(1"), 0/5 



(qtp) 



Sir. 



nq.t 



(g"-i)(g"-2t)...(gr-^)(i«-^) = l 



n — 2\ / in — 1 \ 



(qtp) 



(4.43) 
□ 



The final result of this section is an explicit expression for the analog of (Ve„,p") given by (V^S0 („),p0 (in))B 
(where we recall that P0,(in) = pv^[X + Y]). Observe the similarity with (|4.5p when q = 1/t. Quite unexpectedly, the 
present expression, a priori more complicated, factorizes for q and t generic (recall the discussion surrounding eq. ()4.7p 
pertaining to the non- factorization in the usual case). 

Proposition 21. We have 

(V S0,(„),P0,(i„))b - (g^)(„_i)/2 j ■ (4-44) 

In particular, there follows the two specializations: 

{^U/t'Un).PUv^))B = (^^^^)" (4-45) 

and 

{yq=t=i S0,H,P0,(1"))b = (2n + 1)". (4.46) 
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7B, 



Proof. We first need to obtain the coefficient pi^ [X + Y] in the expansion of V S0_(„) in the p\[X — Y]pfj_[X + Y] basis. 
We have 

[n],^tX + [n + \],.tY = i ([n],,i + [n + 1],,*) {X + Y) + i ([n],,* - [n + (X - F) (4.47) 
and thus this amounts, from Proposition 1 171 to computing the coefficient oi pin[X + Y] in 



1 



\{[n],.t + [n + {X + Y) + i ([n],,i - [n + 1],,*) {X - F) 



{[n]g^t + [n + l]g,t) 



px[X + Y]p,,[X-Y] 



(4.48) 



where we used l|4.22l) . the expansion hn = J2\hnP>'/ (ttSl ^q- 1.2.14']), and the basic property [-''^i^] = Pa [-'^Ipa [i^] • 
Letting A — (1") and ^ = the coefficient of Plr^[X + Y] is easily found to be 



-pir. 



nln.t + \n - 



(gt)(^)zi. 



The proposition then follows from 



Pi 



-Sink 



t+ 



1 /[7l]q,t + [n + l]q,t\" 



2"n! V {qtY'^-^y^ 



{p.^X + Y],p,4X + Y])b = 



(4.49) 



(4.50) 
□ 



Remark 22. As mentioned in the introduction, V^^j_iS0.(„) seems to coincide with Frobj-i ((i?^"), where R^" is a 
certain doubly graded quotient of the coinvariant ring C^". From Remark 1121 one would be tempted to believe that 
there exists for every pair of partitions (A, /x) a bigraded submodule of R^" (isomorphic to the regular module of the 
hyperoctahedral group Bn) whose Frobenius series when q — corresponds to the Schur expansion of Hx,^{x, y; t~^,t). 
This cannot be the case however since V^S0.(„-) — H\^f^{x,y;q,t) is not Schur-positive in general (even when q t^^)- 

Remark 23. The Macdonald polynomials H^'^'^lX] can be defined (up to normalization constants) as the unique basis 
such that 



(i) Hi''^*\x{l^t)]eQ{q,t){s^:pi<X} 



(ii) H['^'\xil-q)]eQ{q,t){s,:^i>\} 



(4.51) 
(4.52) 



The double Macdonald polynomials can also be defined by two similar triangularities. Let (j)t be the plethystic 
substitution 

X^X and {X + Y) ^ {X + Y){l~t) (4.53) 

and recall that ojb is the involution such that ujbsx.ii{x^ y) — Sfj,'_x'{x, y). The double Macdonald polynomials Hx^fi{x, y; q, t) 
can be characterized (up to normalization constants) as the unique basis of the space of bisymmetric functions such 
that 



(i) (j>tHx^^{x, y; q, t) e Q{q, t){s^^,f : (w, i]) < (A, p)} 

(ii) UBO(pqO ubHx,h{x, y; q, t) g Q((j, t){s^^y^ : (w, 77) > (A, fi)} 



(4.54) 
(4.55) 



The ffi'st triangularity is seen as follows: (l)tHx,^i{x,y\q,t) is equal up to a constant to Px,fj,{x,y; q,t~^), which is lower 
triangular in the mx^^, basis and hence in the sa,^ basis (the sa./j basis is lower triangular in the mx^p, basis since it 
corresponds to the specialization q = t = of Px,fj,{x,y; q,t)). Using the symmetry p.l03p . we have immediately that 
Hx^fj,{x,y; q,t) is upper triangular in the sa,^ basis. The two triangularities in the double case are not as symmetric as 
in the usual case due to the noncommutativity of ujb and the plethystic substitution (/>g (whereas in the usual case the 
involution uj commutes with any plethystic substitution). 

Haiman also introduced in |17) wreath Macdonald polynomials. In the special case of the complex reflection group 
G(2,l,n), the wreath Macdonald polynomial 7i^i,{q,t) depend on a choice of staircase partition (5™ (a 2-core) and is 
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indexed by a partition /i of size m{m — l)/2 + n, where n is a fixed integer. They satisfy the triangularities 

H^(9,t)®^(-g)'char(A'[}) eQ(9,<){x'^"°*^^^^ : A > Core2(A) = ,5™} (4.56) 

i 

and 

® ^(-t)'char(A''()) G Olx*^""'^^^^ : A < M,Core2(A) = ,5™} (4.57) 

i 

where [} = C" is the defining representation of G{2, l,n), Quot2(A) is the 2-quotient of A (a pair of partitions of total 
degree n) and Core2(A) is the 2-core of A. This is somewhat reminiscent of our construction, even more so that if 
m is large enough Haiman claims that the corresponding wreath Macdonald polynomials can be given in terms of 
usual Macdonald polynomials. However, we believe that the objects do not coincide given that the symmetry in the 
triangularities (|4.56p and (|4.57p is not present in the double case. 

5. Conclusion: the superspace bridge 

The present construction relies on the formalism previously developed for Macdonald superpolynomials [6j [7] . To 
these superpolynomials, we have associated bisymmetric polynomials by considering the coefficient of a given monomial 
in the S^'s, say 6i ■ ■ ■ 9m for a superpolynomial of fermionic-degree m, and dividing by the Vandermonde determinant in 
the commuting variables xi, - ■ ■ , Xm- We have seen that for sufRcient high fermionic degree, a stable sector is reached. 
In this stable sector, our key result is the product form p.l7p . As a consequence of this remarkable factorization, we 
have been able to prove rather directly a number of properties for these so-called double Macdonald polynomials. In 
particular, we have obtained: 

• The expression for the norm. 

• The expression for the integral form. 

• The evaluation. 

• The positivity and integrability of the Kostka coefficients. 

• The two Macdonald- type symmetry properties of the Kostka cocfhcients. 

In [6l|7], we have presented conjectures related to the above five items but pertaining to generic Macdonald super- 
polynomials. The present results imply that we now have proofs of these results for all cases where to > rt (in some 
sense for roughly half the cases). The precise connection between the present results and our more general conjectures 
is worked out in Appendix C. 

We had a number of mathematical and physical motivations for undertaking the study of the Macdonald super- 
polynomials. One of which, of a combinatorial nature, was to see whether by adding more structure to the usual 
Macdonald polynomials, one could get unexpected new handles on open problems such as a combinatorial description 
of the q, i-Kostka coefficients, generalizing the Lascoux-Schiitzenberger description of the Kostka-Foulkes coefhcients. 
For the stable sector considered here, this particular hope was not fulfilled: our new hyperoctahedral Kostka coefficients 
are roughly sums of products of the usual ones. 

However, we have already presented a conjectural result - pertaining to the non-stable sector, which thereby nec- 
essarily relies on the superspace formalism - that could shed some light on the combinatorics of the usual q, t-Kostka 
coefficients [7]. It says that the simplest superpolynomials, namely those in the to = 1 sector, do provide a refinement 
of the Kostka coefficients. Precisely, it gives a relation between the generalized coefficients KnA{q, t) of fermionic degree 
TO = 1 and total degree n and the usual q, t-Kostka coefficients of degree n -I- 1. 

Conjecture 24. Let A be a superpartition of fermionic degree m — 1, and let H\ be the modified Macdonald super- 
polynomial (see Appendix fCl for more details). Let also tp be the linear application that maps sn to Sq@. Then 

i>iHA)^Hj,s. (5.1) 

This conjecture implies that the usual q,t-Kostka coefficient Kfj,\{q,t) can be calculated from its lower-degree super- 
relatives as 

K^xiq,t)= KnA{q,t) (5.2) 
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where A is any superpartition that can be obtained from A by replacing a square by a circle, and the sum is over all i7's 
that can be obtained from /i by replacing a square by a circle. Moreover, the expression for the sum on the right-hand 
side is independent of the choice of A. We thus relate a Kostka coefficient of a given degree to a sum of lower degree 
Kostka coefficients in the m = 1 fermionic sector, a process that mimics a sort of transmutation of the fermionic variable 
into a bosonic one. 

For example, consider i?(2;i) (2^7 ^5 9) i)- Its Schur expansion reads (using the diagrammatic representation of super- 
partitions introduced in Appendix B) 



Now apply ip : 



(5.3) 



B 



*^^m + (1 + 9^ + «^ + (9 + 9'^) + (9 + 9' + <ft) s 



(5.4) 



which corresponds to the usual Schur expansion of the modified Macdonald polynomials -ff(3 i)(a;; g, t). 

This conjecture can be generalized as follows. Let A and A'^ be two superpartitions such that A'^ is obtained from 
A by replacing a circle by a box. Note that A"^ is not unique for m > 1. Define #(A, A'^) to be the number of circles 
that lie above the row of A where the circle has been replaced by a box. 



Conjecture 25. Let Ha be the normalized modified Macdonald superpolynomial defined as 

Ha = vaHa, with VA^ Y[ (1 -g''Ae('')t'A-(«)+i) 

seA/BA 

and let ip be the linear application that maps: 



'SQ. 



n I a=A0 



Then, 



have: 



MHa) 



(5.5) 

(5.6) 
(5.7) 



n I o=A[=i 



Note the if A has fermionic degree m = 1, this reduces to the previous conjecture since then all the factors v, reduce 
to 1. We stress that A/BA refers to the set of boxes in A that are not in BA, a set denoted by J-'A in Appendix C (see 
below eq. (IC.37[) ). This set is empty when m = 1. 

Remark 26. It is straightforward to check that the application ip is such that ip o ip = 0. In the language of [H], '0 
corresponds to Xide^, the adjoint of the exterior derivative Oidx^- 



Let us make explict the implication of this result at the level of the Kostkas. We have 

^{Ha) =vaY, KnA{q, t)4>{sn) ^ vaJ2 {-l)*'-''^^hA 



A I A=OS 



and 



^(Ha) 



E (-1: 



#(A,r) 



Hr 



#(A,r; 



vr 



E^A, 



r SA, 



(5.8) 
(5.9) 



r I r=As 



r I r=AE 
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By comparing the coefBcients of sa, we get 



n|nE=A r|r=Ai=i 



(5.10) 



This is thus a relation between a hnear combination of Kostkas for fermionic degrees differing by 1. The case m = 1 is 
of course special in that the sum on the r.h.s. reduces to a single term. As already said, when m > 1, can take m 
different values, and therefore the r.h.s. sum contains m terms. Let us then reverse the point of view and see whether 
there are situations for which the Kostkas at fermioinc degree m could be computed from those at degree m — 1. This 
would be the case if the sum on the l.h.s. could be reduced to a single term. This is actually the case when the diagram 
of A has a single removable box. This removable box is thus necessarily the one created when a circle is changed into 
a box. 



Consider an example. Take A = (3; 2), which has one removable box (indicated by x) so that fl — (3, 1; ~ 

D 



A 











X 





(5.11) 



The l.h.s. of (I5.10p becomes ua(— l)_ft'(3 j.) ^(9! Let us set A = (2,0; 2): F in the sum of the r.h.s. of (j5.10p can take 
the two values (1;3,2) and (2; 2,1): 



A 



D 



A^ 



and 



D 



Hence, the r.h.s. of (|5.10l) reduces to if(3;2)(0;3,2)(97 ^) ^ -f^(3;2)(2:2,i)('?i 0- We thus end up with the relation: 

- (1 - g'^t^)i^(3,l;),(2,0;2)(g,t) = K(^3.2)(0;3,2){q,t) ~ ^(3;2)(2;2,1) (?, i)- 

With 

^(3;2)(0;3,2)(g, t) = t + f + qt^ + qt"^ + qh'^ and if(3;2)(2;2,l) (9. t) = ^'(1 + l^t + 9'^' + l^^t^)^ 

we get 

- (1 - g't')i^(3,l;),(2,0;2)(g, t) = -(1 - qh^){t + qf) => i^(3,l;),(2,0;2) (?, t) = t + qt^ . 



(5.12) 

(5.13) 

(5.14) 
(5.15) 



Another instance where this relation appears to be useful is when A has no removable box. In this case, there are 
no such that il^ — A, so that the sum on the l.h.s. of (|5.10l) vanishes. The r.h.s. becomes an identity on alternating 
sums of Kostkas weighted by the factor vr- Here is an example: take A = (3, 2; ), which clearly has no removable box, 
and A = (3, 1, 0; ) 



A 



D 



vrT: (1 - qt) 











3 






o 


'J , (1 - q't^) 







, (1 - q^t) 







D 



This leads to the relation 

(1 - gt)i^(3,2;),(l,0;4)(<Z,i) - (1 " <Z^i')^(3,2;),(3,0;2) (?, t) + (1 " 'z'i)^(3,2;),(34;l) = 0- 

With 



(5.16) 
(5.17) 



-f''(3.2;),(l,0;4)(9,^) =9^^^(l+9^0 and -?f(3.2;),(3,0;2)('?, = -f'^(3,2;),(3,l;l) =^(l+9^*)> 



(5.18) 



this is easily checked to be satisfied. 
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Appendix A. Monomials and power-sums in superspace in bisymmetric form 

Proposition 27. The bisymmetric monomial deduced from the supermonomial of fermionic degree m is given by 

mx^^{xi,...,XN) = sx{xi,. . . ,Xm)mf,{x.m+i, ■ ■ - Xn), (A.l) 

where S\ and m\ are respectively the Schur polynomials and the usual monomial functions. Similarly, the bisymmetric 
power-sum reads: 

PX,fxixi, . . .,Xn) = Sx{xi, . . . ,Xm)Pfj.ixi, . . ■,xn)- (A. 2) 

Note that in the first case the two functions on the rhs depend upon distinct set of variables, which is not the case in 
the expression of the power-sums. 

Proof. Let us first establish (jA.l[) . starting with the expression for the super- monomial: 

TOA = y^ 51 ^-^i-"^™^'^' /a^(1) =n"A=(*)!, (A.3) 

with n\s(i) being the number of occurrences of i in A^, and m is the number of fermions. The permutation JC^j 
interchanges the pairs {xi,9i) and {xj,9j). Focusing on the term of mA that contains 9i ■ ■ ■ 0m, we have 

[6i-- ■ 0jn] niA = -r-^-7-TAl,...,raSm+l,....NX^ (A. 4) 

jAH-'-j 

where A is the anti-symmetrizer operator (which, here, acts on the variables xi ■ ■ ■ Xm) and S is the symmetrizer 
operator (acting on the variables x^+i ■ ■ ■ ,xn)- The coefficient /a=(1) ensures that repeated terms count for 1 in the 
expression. Since A and S are independent operators (acting on different sets of variables), we can write: 




(A.5) 



where e{w) is the sign of permutation w and Kij interchanges Xi and Xj . By dividing this expression by the Vandermonde 
determinant in the variables xi - ■ ■ Xm, denoted A^, and using the decomposition A — (A"; A*) = (A-|-^™; /i), we obtain: 



TOA,^ (A. 6) 



where x denote the variables xi, • • • , Xm- The term in the first parenthesis is nothing but the definition of a Schur 
function in the variable xi ■ ■ ■ Xm, that is s\{xi, . . . , Xm)- The second term is simply the monomial function in the other 
variables over the partition fx, that is, m^(a;m+i, . . . ,X]\[). We have thus recovered (lA.ll) . 

For the derivation of (|A.2p , we proceed in a similar way. The complete power-sum superfunction labelled by A reads 

PA = PAJ • • • PA^PAf ■ ■ ■ PA^_,„ (A.7) 
where pk = J2iLi ^i^i ^^-^ Pk — J2iLi ^i- Taking the coefficient in 9i ■ ■ ■ 6m we have : 

[6*1 • • .6*™] PA = [Al,...^mX^ ] PA={xi, ■ . ■,xn)- (A. 8) 

Divide this by the Vandermonde in the m first variables and we finally obtain 

1 /E™es £(,w)K^{x^"'+^)\ 

[6i--- 6m] -J— Pa = — jT-^ — - — ^ — P^l{xi, . . . , xn) =: Pa,/. (A.9) 

^™ y \.\.i<i<j<my'^i ^j) ) 

where the term in parenthesis is the Schur functions s\{x\, . . . ,Xm)- This gives (jA.2p . □ 
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Appendix B. Induced properties on pairs of partitions: conjugation and dominance order 



As mentioned in the introduction, a superpartition is a pair of partitions of the form A = (A"; A'') where the partition 
A" has m distinct parts (the m-th one can be equal to 0). We define 

A*=A''UA'' and A® = (A'' + 1'") U A^ (B.l) 

It is manifest that the pair of partitions (A®, A*) fixes the superpartition A. A diagrammatic representation of A is 
obtained from the Ferrers diagram of A® by changing into circles the cells of A® /A*. For instance, if A = (3, 1, 0; 2, 1), 
we have 



A® 



A* 



(B.2) 



which gives 



A 



D 



(B.3) 



Recall that we can associate to a superpartition A of fermionic degree m a pair of partitions A and /i in the following 
way: 

A = (A'^; A'') o (A, fi) = (A'^ - (5", A"). (B.4) 
When m > n, this establishes an obvious bijection between superpartitions (A°;A'*) of fermionic degree m such that 
|A"| + lA**! — n + m{m — l)/2 and pairs of partitions (A, /i) such that |A| + = n. Before describing how the 
conjugation and the dominance-ordering properties are induced from superpartitions to pairs of partitions, we establish 
some elementary results that will be used for this analysis. In that regard, it is convenient to first introduce a convention 
concerning the positions of the boxes corresponding to A within the diagrammatic representation of the superpartition 
A. We choose to place the boxes of A (the boxes marked with a o in the example below) in columns that are not 
fermionic (that is, that do not end with a circle). For instance, consider A — (2, 1), ji = (3, 1) and the corresponding 
A for m = 3, 4, 5 (all in the non-stable sector, illustrating the fact that the two partitions get disentangled before 
m > n — 7): 





o 




o 


• 


• 


• 






o 


'J 




• 









D 



D 









o 




o 








o 




• 


• 


• 














-■ \ 






• 









D 



(B.5) 



Lemma 28. Let \,fi,A and m be defined as in (jB.4p and suppose that m > = |A| + If £{X) ~ i then the l-th 
entry o/ A" is strictly larger than the first entry of A'^ , that is, 

A1>Al=fii. (B.6) 

In particular, in the diagram of the superpartition A, the cells marked with a o appear strictly above those marked with 
a •. 



Proof. The condition m > n implies 
Hence 



m > Xi + ■ ■ ■ + Xg + fii > i + Hi > £ — + fii 
Ag ^ Xe + m - e > fii = AJ. 



(B.7) 
(B.8) 
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□ 

The bound m > n is not the optimal one ensuring the separation of A and in A but it is sufficient for our purpose. 

Lemma 29. In the diagram of the superpartition A, when m > n, the cells marked with a o appear strictly to the right 
of those marked with a • . 

Proof. By inspection, we see that the column in A where the o corresponding to the first column of A are inserted is 
the m — X[ + l one. The lemma will then hold if m — A'j^ + 1 > fii. Given X[ — £{X) ~ £, this corresponds to m > £ + /ii, 
which is a consequence of m > n as seen in (jB.7p . □ 

We now come to the conjugation property. 
Lemma 30. Suppose that as in (|B.4p . we have A <-> (A, /i) with m > n. Then 

A' o (//, A') . (B.9) 



Proof. This follows directly from the definition of the conjugation for superpartitions, obtained by the interchange of 
rows and columns. □ 

An example will make this completely obvious: consider the pair (2), (3, 1), so that, with m — 6: 

D 













o 


o 










'J 








:) 




• 


• 


• 




















• 









A^A' 



(B.IO) 



o 

We observe that our convention of placing the boxes of the first partition into bosonic columns of A is preserved by the 
conjugation operation. 

Recall that the dominance ordering on bi-partitions was defined in (jl.l4p . In the following lemma we give three 
equivalent form of the second condition in the dominance ordering on bi-partitions. For this purpose, it will prove 
convenient to relax the condition |A| = in the dominance ordering and say that 

\> H if Ai H \-Xj> Hi + h /ij Vj, (B.ll) 

even in cases where |A| 7^ Observe however that the equivalence 

A > A' < (B.12) 

only holds if |A| = 

Lemma 31. Suppose that |A| > |a;|. Then the following three statements are equivalent: 

(1) |A|+/^> |c^|+r; 

(2) Ai'U < J]' 

(3) f^' < V' 

Proof. We have that (1) and (2) are equivalent since 

|A| + Ml + ■ • • + /ij > 1^1 + ??i + ■ • • + Vj Vj ^ M + - 1^1) > 77 ^ m' U (ll^l"!"!) < tj' , (B.13) 
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where we stress that /j, + (|A| — |w|) stands for the partition (/ii + |A| — |w|, /i2, 1-J.3, . . .). It is immediate that (2) impHes 
(3). Finally, (2) follows from (3) since (fl'^l^l'^l) is dominated by any partition (and in particular by the partition 
H hr/^+i - \n\,r]'e^2:Ve+3:- ■ where i = £{fi'j) . 

□ 

The following proposition was essential to deduce Theorem [5] from Theorem [U and thus to connect the Macdonald 
polynomials in superspace to the double Macdonald polynomials. 

Proposition 32. Suppose that A o (A,/i) and fl o with m > n = \X\ + = + \ri\. Then 

{X,fi) > iLj,r]) <^ A>Q (B.14) 



Proof. For the purpose of this proof, we first modify our convention for the insertion of the boxes of A and /i within 
A into a prescription that describes A* built from the core S™. Reconsider example (jB.sp but now with m — 7 and 
identify the o and • as the upper and lower boxes, respectively, that lie outside the sub-diagram (5™. This yields: 



D 



A* 



(B.15) 



o 

where the unmarked boxes in A* are those of 5^. More generally. A* can be obtained by adding A (resp. /i') to the top 
rows (resp. leftmost columns) of 5™. In other words, 

A* = ((5™ + + A. (B.16) 

It is clear from the above diagrams that the row of A that corresponds to the first row of fi (namely, the (m + 1 — /^i)-th 
row) is also the highest row in A* containing a •. Therefore Lemmas [28l and [29l implv that if m > n and A o (A, /i), 
then A* can always be described as above, in particular, with the o and • separated from each others. In the context 
of this proof, A* will always stand for the diagrammatic representation (|B.15|) . The following elementary observation 
will be fundamental. 

OBS 1: Suppose that the highest • in A* lies in row r and column m + I — r (the highest • is always alone in its row 
by construction). If i < r, then the number of o's strictly below row i is not larger than the number of rows 
(r — i — l) between rows i and r. Similarly, if i < r, then the number of o's strictly to the left of column m + l — i 
is not larger than the number of columns (r — i — l) between columns m + l — i and m + 1 — 



Ml 



Xe, where 
- 1 



£{X). The statement is 
(B.17) 

(B.18) 



This is seen as follows. The number of o below the i-th row of A* is A^+i 
that 

Ai+i + -- - + A^<r — i— 1 = TO + 1 

or equivalently, that 

fj.1 + i + A,;+i + ■ ■ ■ + Xe < m. 
But since 

Hi+i + A,+i H h A^ < + |A| = n < TO, (B.19) 

the result follows. The column case is treated in a similar way, the number of o at the left of the column m + l — i 
being < 1 + Aj+2 H h A^. 

First, we show that given our hypotheses, the following implication holds: 

(A,^)>(w,r/) =^ A>n. (B.20) 
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The main step towards that goal is to estabhsh that 

{X,fi) > {u,r]) =^ A*>n*. (B.21) 
We proceed by contradiction. Suppose that (A, /i) > {lu, r/) and A* ^ $7*, and let i be the highest row such that 

AI + ---A* <ni + --- + n*. (B.22) 

Note that given these conditions, we have necessarily ft* > A*. 

From the first condition in ()1.14|) and the construction (jB.lsp of A*, we can conclude that i > £{uj). Suppose that i 
lies above the highest • in A* (which we assume lies in a certain row r) . Let d be the number of o in A* strictly to the 
left of column m + 1 — i, so that d < A^+i + • • • + < |A| — \uj\. Since d < |A| — we can use Lemma [5T] to deduce 

A^'i + • • • + fJ-'t., + 1 + • • • + 1 = +d<7][ + ---+ rj'^^+^ . (B.23) 

On the other hand, we have d < r — i — l = m — fj,i — i by OBS 1, which implies that the length (fii + d) of the partition 
/i' U l"* is smaller than m + l — i (and thus the ^'s corresponding to t][, . . . , r]'i_ii+d below row i). From ()B.22p and 

|A*| = \ft*\, we also get that below row i there are more cells of A* than cells of ft*. Therefore 

l/i'l + d > number of cells of rj below row i > ti[ + ■ ■ ■ + v'ni+d ^ 

(B.24) 

which leads to a contradiction (compare (|B.23I) and (IB.24p ). 

Now suppose that i does not lie above the highest • in A*, and let £ = A*^^- In this case, due to the hypothesis 
(A, /i) > (which implies in particular the third expression in Lemma [5T|) . we get 

^i[ + ■■■ + ^i'e<v'l + ■■■+Ve■ (B.25) 

According to the working hypothesis we want to contradict, namely il* > A*, there must again be below row i more 
cells of A* than of il*. Hence (letting I - {i + 1) ^ s) 

^[ + ■■■+11'^ — s{s — l)/2 > number of cells of f] below row i > ri[ + ■ ■ ■ + i]'^ — s{s — l)/2 , (B.26) 

where s{s — l)/2 is the number of cells of f]' and fi' above row i + 1 up to column £ (note that the condition ft* > A* 
ensures that this number is the same for ^' and ry'). Comparing (jB.25|) and (jB.26|) leads again to a contradiction, and 
we can conclude that (jB.2ip holds. 

Finally, observe that 

A* = (A + (5") U A* and A® = (A + (5" + 1™) U a* = (A + 5"+^) U , (B.27) 

that is. A® corresponds to A* with m replaced by m + l. But (jB.21[) also holds for m + 1 (it holds for all rn > n), 
which means that 

(A, n) > {lu, r/) =+ a* > n* and A® > n® (B.28) 

and ((B^O)) follows. 

We now need to show the reverse implication: 

{X,fi) > {Lu,ri) ^ A>n. (B.29) 

It is sufficient to show that given our hypotheses 

(A,A*) > (w,?7) -4= A® > 17®. (B.30) 

First we show that A® > fl® implies A > w. From (IB.27p . we get that the diagram of A® is that of A* with m replaced 
by TO + 1. From this construction, we have immediately (using £ = £{X)) 

A® >n® => \i + --- + Xe>uji + ---+uje (B.31) 

Hence Ai + ■ ■ • + Ai > uji + ■ ■ ■ + uji will hold for all i if |A| > |aj|. Suppose that |a;| > |A| and let k = \X\ + 1. An 
observation similar to OBS 1 will prove useful. 

OBS 2: Suppose that the highest • in A® lies in row r. Then the number of o's in A® is not larger than r — 2. 
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Indeed, the number of o is equal to |A|, and in this case r = m + 2 — /ii. Thus |A| < r — 2 = ni — fii since 
|A| + /ii < |A| + \fi\ < m. It is then immediate that the highest • in A® lies below row k since OBS 2 gives |A| < r — 2, 
or equivalently, r > |A| + 2 > fc (that is, the cells of /i do not contribute to A® up to row k). Using 

uJi + ---+ujk>k=\X\ + l, (B.32) 

which follows from the hypothesis \Ld\ > |A|, we then get 

nf + --- + n® ~k{m + l) + k{k + l)/2 = cji + • • • + t^fc > |A| + 1 > A® + • • • + A® - fc(m + 1) + fc(fc + l)/2 (B.33) 

which is a contradiction to A® > il®. We thus have A > a;. By conjugation, we also have rj > fi, and thus (jB.30p holds 
by Lemma [3TJ □ 



Appendix C. Proofs of the conjectures in the stable sector 

In this section we recall some of our previous conjectures concerning the Macdonald superpolynomials and show that 
in the stable sector they match statements that were demonstrated in this article. These conjectures are thus partly 
validated but we stress that they preserve their conjectural status in the non-stable sector. 

Let us first recall that the Macdonald superpolynomials P\{x,9) — P\{x,9;q,t) are defined as in Theorem[T]by the 
two conditions 

PA(a;,6';9,t) = mA(x, 6*) -flower terms, and {{Pa{x,9), Pn{x,9)))q^t 5aq 
but with the scalar product for the bisymmetric power-sums replaced by its superspace form: 

{~qPh{Px,^.,Pu,n))g.t = {{pA{x,e),pn{x,9)))q,t (C.l) 

where m is the fermionic degree of A. This matching factor (which comes from gl^"!^!-^!) plays no role in the orthogonality 
conditions, and affects only the value of the norm. 

C.l. Norm and integral form. We first discuss two conjectures related to the norm and the integral version of the 
Macdonald superpolynomials. These are as follows. 

Conjecture 33. [6J The norm oj the Macdonald superpolynomial P\(x,9;q,t) is given by 

iPA{x,9),PA{x,9))),.t = (-l)(")g|A"i:j|M, (C.2) 

where 

4(9,*)= n (1-9""®''^*'"**'^^')' h\{q,t) = hi,{t,q). (C.3) 

seBA 

In the previous equation Bh. stands for the subset of boxes in the diagram of A whose row and column do not both end 
with a circle. This is referred to as the set of bosonic boxes of A. For instance, in example (|B.2p . the bosonic boxes are 
(1,3), (2,1), (2,2) and (4,1). 

Conjecture 34. [6 The integral form oJ the Macdonald superpolynomials P/^(x,9;q,t) reads 

Ja(x, 9; q, t) = 4(g, t) Pa(x, 9; q, i), (C.4) 
where hj^{q,t) is defined in (|C.3P . In other words, the monomial expansion coefficients of J\{x,9;q,t) belong to "Llq^t]. 

In the following, we will make the connection between the conjectured formula (|C.2I) of the norm of the Macdonald 
superpolynomials and the formula p.25p giving the norm of the double Macdonalds. In view of establishing this 
equivalence, we recall that the two norms differ by the factor (— 
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Lemma 35. In the stable sector, the super and double norms are related by the equation 



h\{q.t) 



(C.5) 



where A = (A + 5"^; /i) with m > n = |A| + 



We observe that the r.h.s. of (|C.5[) is m-independent (it is a formula that pertains to the stable sector) while this is 
not obviously true for the l.h.s. So we first establish this fact. 

Lemma 36. Let A = (A + (5™; /i). Then h'j^lq, t), defined in (|C.3[) . does not depend upon m when m>n=|A| + |^|. 

Proof. When m > n, the boxes of A that belong to the subdiagram BA are those corresponding to the inserted parts 
of A and in A (see Appendix [B]) . For example, with X, n — (3, 1, 1), (2, 2, 1) and m = 10: 



BA 



D 



(C.6) 



O 

where the boxes marked with an x are not in BA (they lie at the intersection of a row and a column ending with a 
circle). Now, in the expression for h^{q, t), the product over the boxes of BA can manifestly be separated into a product 
over the bosonic boxes of the following two smaller superpartitions: 

n := (A + ,5^(^); ), and 

In the above example, these are 



A:- (<5^-m)- 



(C.7) 



n 



o 


X 


X 


o 


o 


o 


X 








o 











A 



(C.8) 



O 



The decomposition takes the form 

hA{q,t)= ]J (l-g°n® (^)+i) Yl (l-g''Ae(^')i'A*(^')+l). (C.9) 
seBfl s'gBA 

This is a simple consequence of the fact that the leg-length and arm-length of the corresponding boxes do not depend 
on the rest of the diagram (see Corollary [29]) . Now, the m- independence of both ft and A entails that of h^{q,t). An 
identical result holds for h\{q,t). □ 

We now turn to the proof of Lemma 1351 

Proof. The objective is to rewrite the Cl and A contributions in (|C.9p solely in terms of the A and fi data, respectively. 
The first step amounts to reorganize the terms in the first product as follows: 

gOn(S)(s)^;n.(s) + l _ ^Ofjg,(s)-in.(s)-l|-^^yfi.(s) + l^ (C.IO) 
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Since all the parts of f2 are fermionic, aQ®(s) — 1 = aQt{s). Note also that the boxes of BCI correspond exactly to 
those of A. Then, it is easy to see that lfit{s) — l\{r) where r is the box corresponding to s in A. Similarly, we 
have af2* (s) — Zfj* (s) = a,\{r), the subtraction removing precisely the contribution to the arm of the non-bosonic boxes 
belonging to the staircase partition. We thus have: 

Yl (1 = (C.ll) 

sGBfl rex 

Similarly, we reorganize the contribution of A in (|C.9|) as follows: 

^aAe(«')i'A*(s') + l — (g^)a^®(s')^'A*(s') + l-a^®(s')^ (C.12) 

Since the rows of fi, when inserted into the staircase partition, constitute rows by themselves (non- fermionic ones), we 
obviously have a^®(s') = a^j,{r') with r' the box of /i corresponding to s' . Arguing as before, we also have l]^'{s') — 
ax@{s') = lfi{r'), so that 

Jl (1 _ q<^A@(^')t'A''-'')+^) = n (1 ~ (C.13) 

Collecting these results yields 

hiiq,t) = 11(1 - Jl (1 - iqtr-<^^'h'-(^"^+') = c^{q, qt)c,{qt,t). (C.14) 

A similar expression holds for h\{q^t)\ 

h\{q, t) = hi, {t, q) = V 9i)cA' {qt. q) (C.15) 
where we used A' ~ {fi' + 5™; A') (see Lemma [50)1 . The expressions for the norm can thus be related: 

hi{q,t) c\{q,qt) c^,{qt,t) 

□ 

Lemma |35] readily implies that: 
Corollary 37. Conjecture is true for m > n. 

On the other hand, the relation (|C.14[) implies the equivalence of the integral forms (|C.4p and p.76p : 
Corollary 38. Conjecture \34\ is true for m > n. 

C.2. Kostka coefficients. Let (p be the endomorphism of Q{q,t)[pi,p2,P3, ■ ■ ■ ;po,pi,P2, ■ ■ ■] whose action on the 
power-sums is 

V{Pn) = -^Y^—jPn and (p{Pn)=Pn- (CIT) 

We then define the modified Macdonald polynomials in superspace as 

Ha{x, 9; q,t)=^ {Ja{x, 9; g, t)) . (C.18) 
We also introduce the Schur superpolynomial as the limit q ^ t ^ oi the Macdonald superpolynomial 

SAix,9)^PA{x,9;0,0). (C.19) 
Conjecture 39. [GJ The coefficients Ki2A{q,t) in the expansion of the modified Macdonald superpolynomials 

Ha(x, 9- q,t)^Yl ^) (C-20) 

o 

are polynomials in q and t with nonnegative integer coefficients. 
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When m > n, the Kostka coefficient KnA{q,t) is equai to the coefficient KK,,'y x,p,{q,t) defined in (|1.19|) (where 
A o (A, fi) and ■(-> (k, 7)) since the transformation of the previous equation into its bisymmetric form does not affect 
its expansion coefficients. Therefore, as a direct consequence of Proposition we have: 

Corollary 40. Coniecture \39\ is true for m > n. 



Finally, the following symmetries of the coefHcients KQ_/^{q,t) have been observed [71 Section 7.1]: 
KnK{q.t) = Kn'K'it.q), and KnK{q.t) = g«(A')^n(A) AW(g"\t"'), 
where fi(A) is given by 



n(A) = n{Sk) - d^(A) 



with 



n{\) 



1)A, 



(C.21) 



(C.22) 



In the previous equation, Sh. is the skew diagram SK = A®/(5'"+^ (we consider that n{\/ ji) = n(A) — n{fi)). The term 
d^{A) is defined as follows: fill each square s G BA (we recall that BA was defined after Conjecture 1551) with the number 
of boxes above s that are not in BA. Then, d'^{A) is the sum of these entries. For instance, considering example (jB.2p . 
and marking again by an x the boxes in A that are not in BA, we have 





X 


X 







1 


1 




^'^((S, 1,0; 2,1)) ===4: 


X 


-■ \ 






2 







(C.23) 



As previously mentioned, in the stable sector Kfi\{q, t) — i^K,7 A,/^(5, t) with A = (A + 5™\ fi) and 51 — (k + 5™; 7). 
The symmetries (jC.21[) must thus, in the stable sector, be a consequence of Corrolarv ll3l This is immediate for the 
first one. In the following lemma, we show that this is also true for the second one by establishing the equivalence of 
the expressions n{A) and n(A, ^) (the latter defined in Corollarv [T5| when m > n. 



Lemma 41. For A = (A + (5™; fi) and m > n = |A| + we have 

n{X) + 1^1 + Ti{p) + n{p') = n{SA) - d{A) 



(C.24) 



Proof. First, consider the expression n{SA). Since SA = A®/(5'"+^ = ((A + (5™+^) U ^)/5™+\ the expression n{SA) 
is the "depth-weighted sum" n{-) (referring to i — 1 as the depth of a box in row i) of the boxes of the diagrams of A 
and fi when inserted in the staircase partition (m, m — 1, . . . , 0). For the analysis of n(SA), it is convenient to view the 
insertion of the diagrams of A and ^ into that of (5™+i to be as described in the proof of Proposition 15^ in Appendix [Bl 
Consider again the example oi X, 11 — (3, 1, 1), (2, 2, 1) and m = 10. We have: 



SA 



(C.25) 



The boxes marked with an x are not considered in the evaluation of n{SA). Clearly, we have 

niSA) ^ n{\) + GM 



(C.26) 
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where Gm{p-) is some contribution that depends on the diagram of /i and on m. For a general partition fi inserted into 
gm+i (^fj-oni the bottom), we can compute Gm{^) as follows. First observe that the boxes of /i are ordered as 



Gm(lJ.) 



(C.27) 



The first box 171 is located in the (to + l)-th row of the total diagram. Consequently, it has depth gi = m. Similarly, 
we have: 



g2 = (m-l), g3 = {m-2), g4 = (TO + l), g5 = (TO + l-l), ge^{m + 2), 
One easily sees the general pattern: 

G„(/i) = (to + 1 - 1) + (to + 1 - 2) + . . . + (to^ + 1 - ^1) 
+ (to, + 2 - 1) + (to + 2 - 2) + . . . + (to + 2-^2) 



(C.28) 



+ (to + e{^i) - 1) + (to + £ifi) - 2) + . . . + (to + £{ii) - 

Mi 

1=1 j=l 

This summation is readily evaluated: 

i i 

where we used the relation f^iif^i ~ l)/2 = n{fj,'). We have thus 

n(5A) = n(A) + n(^) + to|^| - n(/i'). (C.31) 

Let us turn to the contribution of rf'^(A). For its computation, we need to return to the standard insertion of A and 
/i within A illustrated in (jB.51) . Now, d^{A) can naturally be split in two parts 



d^(A)=df(A) + df(Ai), 



(C.32) 



simply because all the boxes of BA are those corresponding to the inserted ones of A and /i. Clearly, no box of the 
subdiagram A/BA lies above the boxes corresponding to A in BA since the latter are all inserted in the topmost rows. 
Consequently, we have df (A) = 0. Now, consider the first row /ii of fi. This row is inserted in the (to — ^1 + l)-th row 
of A (see (jB.Sp and (jB.lOp for instance). Each box of this row of length /ii has thus exactly m — fii boxes of A/BA 
above it, for a contribution of (to — ^i)(/ii) to (m)- For the next row of /i, two cases are possible. If /12 = Mii this 
second row of is just below the row /ii in the diagram A, in which case its contribution to (if (m) i^ ("^ ~ M2)(M2)- 
Otherwise ^2 < Mi, so that this row /i2 is inserted in the (to — /i2 + 1 + l)-th row of A. Therefore, the number of boxes 
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of A/BA that lies above each box of this row is (?7i — /i2 + 2 — 1 — 1), since we have to remove the box belonging to row 
jii. Its contribution is then also (to — /^2)(/^2)- In general, the contribution the part to df (a*) is 

[(position of the row /i^ in the diagram A ) — 1 — ( ^ of rows of fi above /ife)] x (p-k) 

= ((m - fik + k) - 1 - (k - 1))(m/c) = (to - ^fe)^fc. (C.33) 

We thus have 

d^(A) = d^ifi) = ^(m - ^{m- 1)|mI - Mf^')- (C.34) 



All together, this gives: 



n{SA) - d'^iA) = n{X) + n(/i) + |^| + n(/i'). 



(C.35) 

□ 



C.3. Evaluation. In [6], we have formulated an intriguing conjecture for the evaluation of the Macdonald superpoly- 
nomials. Let F{x, 9) be a polynomial in superspace with fermonic degree to and suppose that N > m. The evaluation 
of F{x, 9) is defined as 

'dg,^---de,F{x,9y 



-N,i 



^[Fix,9)] :-- 



where the values Ui are given by 



A{xi,...,Xm) 



Xi—Ui,...,Xj\f—Un 



(C.36) 



(C.37) 



* ginax(m — i.O) ' 

and where A{xi, . . . ,Xm) = ni<i<j<m(-^* ~ ^j) Vandermonde determinant in the variables xi, . . . ,Xm- The 

conjectured formula involves the quantity d-^{A) defined as follows. Fill each square s e J^A (these are the boxes whose 
column and row both end with a circle) with the number of boxes above s that are not in J^A. Then, d'^{A) is the sum 
of these entries. For instance, considering again the example (jB.2p . and marking by x the boxes in A that are not in 
J^A, we have 



(i-^((3,l,0;2,l)) = l 









X 


X 


X 




1 


'' ^ 




X 







(C.38) 



O 



Conjecture 42. Let A be of fermionic degree to and suppose that N > i(A®). Then, the evaluation formula for the 
Macdonald superpolynomial P\{x,9;q,t) is 



EN^m[PAix,9;q,t)] = 



n 1 



^(™-l)|M/5".|-„(AV*")4(5,i) ^^^^ 

where h^{q,t) is defined in (IC.3p . while a'{s) and l'{s) are the arm colength and leg colength defined in 

We now show that in the stable sector, this evaluation formula agrees with the one given in p.69p . 
Lemma 43. For A — ((5™ + A; fi) and to > n = |A| + \fi\, we have 

F^N,m{P\.fi{x,y;q,t)) = EN,m[PAix,9;q,t)]. 



(C.39) 



(C.40) 



Proof. The product over SA in (jC.39|) can be separated into two contributions: the boxes of the diagram of A and those 
of the diagram of ^ (see for example (jC.25|) '): 

n (l-q<c.iskN-l'^sis)^ Yl (l^q<c.is')t^-l',,A^')y (C.41) 

se(5'"+i+A)/5'"+i s'e((5"+i+p)75'"+i 

For the product corresponding to the diagram of A, we have a^g,(s) = to — I'^is) + a'j^(s)since a box in row i of A 
has (to — I + 1) extra boxes to its left when considered within SA (and l'^{s) — i — 1). Because there are no boxes 
above the rows of A, we have ^^@(s) = I'xis)- for the product over the boxes of fi, observe that the topmost box 



FROM MACDONALD POLYNOMIALS TO THEIR HYPEROCTAHEDRAL EXTENSION: THE SUPERSPACE BRIDGE 



39 



s of each column of /i (into SA) has exactly m — boxes above it (i.e., m — j + I if s is in the j-th column and 

a'{s) = j — 1). Thus, for s' E {S"^'^^ + we have l'^@{s') — ^^(s') + m — a^(s') and since there are no boxes 

to the left of the cells corresponding to the diagram of fi in SA, we can write a^g,(s') — a^(s). With A°-/S™ = A and 
using h^{q,t) = c\{q,qt)c^j^{qt,t) which holds true in the stable sector (cf. (jC.141) '). we have: 

j-n(5A)+d-^(A) 

(C.42) 

Now consider the quantity (i-^(A). Clearly, it will only depend on the superpartition A defined in (|C.7p . In A, there are 
/ii — 1 fermionic rows. Every box in a fermionic row contributes equally to 'i'^(A) and this contribution is the number 
of parts in n greater than this fermionic row. This entails the following expression: 

Ml Ml 
d-^(A) = ^(i - 1)[ multiplicity of parts > i into ^ ] ^ ^(z - 1)[ Card{j : /ij > i}]. (C.43) 

i=i i=i 

But since Card{j : jij > i} corresponds to /i^ and fii — ^(^')i have that (i'^(A) — n{(j,'). Using this result and the 
expression (|C.31|) for n{SA), we thus get: 

n{SA) + d^{A) = n{X) + m|/i| + n(^). (C.44) 

Substituting this result into (jC.42l) . we finally obtain: 

Eat,™ [Pa (a;, 6*; q,^)] = ^(m-i)|A| 11 i ^ qa>.{s)(^qt)ix(s)+i 11 r~(^t)M^OiU^O+i 

~ q{-m-l)\\\ 11 1 _ qaxis) (^q^y ^(s) + l 11 1 _ (gi)a„(s')t'M(''') + l 

t^M „ (gt)''A(«) -g°'A(^)(g"'^^) „ (^)C('^') - (gt)%(''')(t^-'") 
~ g(m-l)|A| 11 1 _ gaA(s)(giyA(s) + l 11 r~(9ijMs0iU^0 + l 

= g(^-i)|A| "^ACg"'^"^; gt)7i;^(^^-"'; gt, t) (C.45) 
which is precisely the formula for ^N.m{Px,^^{x, y;q,t)). □ 
Corollary 44. Conjecture \42\ is true for m > n. 



Appendix D. Tables of Kostka coefficients 
Table 1. K\^^i^^^{q,ty for degree n = 1. 





(i),0 


0,(1) 


(1),0 


1 




0,(1) 


t 


1 



Table 2. K\^^i^^^{q,ty for degree n = 2. 





(2),0 


(1,1), 


(l),(l) 


0,(2) 


0,(1,1) 


(2),0 


1 


<3 


9 + 9^ 


9^ 


9^ 


(1,1), 


qt 


1 


9 + l^t 


^'t 


9^ 


(1),(1) 


t 


t 


1 + 9t 


9 


9 


0,(2) 






t + gi^ 


1 


9* 


0,(1,1) 






t + 


i 


1 
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Table 3. K\^^ ^^i,{q,ty for degree n = 3. 





(3),0 


(2,1),0 


(2),(1) 


(1^),0 


(1,1), (1) 


(1).(2) 


0,(3) 


(1),(1,1) 


0, (2, 1) 


0, (1^) 


(3),0 


1 


q + q'^ 


q + q' + q'' 


q' 


q'+q'+ q" 


q-'+q^+ q" 


q' 


9^ + 9* + 9=' 


9^+9'' 


9-^ 


(2, 1),0 


qt 


1 + q't 


q + q't + q-U 


q 


q + q' + q''t 


q-' + q-'t + q'^t 


q't 


9^ + 9-' + q^t 


q-' + q'-'t 


9" 


(2),(1) 


t 


t + qt 


1 + qt + q'-'t 


qt 


q + qt + q'-'t 


q + q' + q't 


q' 


q + q' + q-'t 


q'+q' 


9^ 


(1-'),<1 


q^t^ 


qt + q'-'t'-' 


q't + qH'^ + q'^t^ 


1 


q + q't + q^i' 


qH + q'^i' +q=t'' 


q*'t-> 


q' + qH + q-^t' 


q^t + q^t' 


9^ 


(1,1)>(1) 


qt' 


t + qt' 


2qt + q't' 


t 


l + qt+ q't' 


q + q't + q-'t' 


q-'t 


9 + 29^i 


q'+q-'t 


q' 


(1),(2) 


t' 


t'-' + qt'' 


t + 2qt' 


qt' 


t + qt'-' + q'-'t'' 


1 + qt + q-'t' 


q 


29* + q-'t-' 


q + q^t 


q't 


0,(3) 




qt^ + q't^ 


t'^+qt-'+q't^ 


q-h'^ 


qf + q'f^ + q^'t" 


t + qt'-' + q-'t-' 


1 


qt' + q't^ + q-'t'^ 


qt + q't' 


q-^t-i 


(1),(1,1) 




t'+t-' 


t + i' + qt-' 


t' 


t + t'-'+ qt'-' 


t + qt + qt'-' 


qt 


1 + qt + qt'-' 


q + qt 


q 


0, (2, 1) 




f' + qt'-' 


t' +t^ + qt'^ 


qt" 


t' + qf' + qt'^ 


t + t'-' + qf' 


t 


t + qt'-' + qt-' 


1 + qt' 


qt 


0,{1-') 




t'^ + t'' 


t-^ + t* + 


t' 


t' +f> +t'^ 


+ + t* 


t' 


t + t'' +f' 


t + t' 


1 
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